WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
A unified model predicts future video and query-based 3D trajectories together. Trained on 1.44 million samples, it has the lowest ADE and FDE among tested models.
The claim
One model predicts future video and query-based 3D trajectories in a single backbone.
The problem
Video world models leave 3D dynamics implicit.
Action-based methods depend on specific robot configurations.
Flow-only 3D models learn from a narrow signal.
How it works
3D motion becomes continuous displacement tokens on the video grid.
A single mixture-of-transformers backbone processes video and motion.
Rectified-flow training denoises both modalities together.
Pretrained video models supply physical commonsense.
Training uses 1.44 million simulated and real object samples.
Figures
What changes
WorldFlow has the lowest ADE and FDE on all three test sets.
On SpatialCog it reaches 0.060 m ADE, 54.2% below the best baseline.
Against its flow-only variant, ADE falls 55.9% and FDE falls 68.5%.
Limits
Evaluation uses three sets with 1,678 benchmark samples.
The abstract reports no runtime speed or memory figures.
The benchmark draws from 11 simulated and real data sources.
Source · arXiv