arXiv cs.AI· Yuhao Lu, Jiongxu Chen, Hexiang Wei, Jiajin Wu, Juelin Li, Mingjian Liang, Fan Yang, Zhuowei Liang, Zihao Liu, Zhennan Wang, Sheng Wang, Lin Shao·· 3 小时前AI 评分42
WorldFlow：通过大规模预训练实现视频与 3D 运动联合预测
WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
AI 导读
WorldFlow 是一个生成式世界模型，通过大规模预训练联合预测未来视频和基于 query 的 3D 轨迹。它将 3D 运动参数化为与视频网格对齐的连续位移 token，用统一的 Mixture-of-Transformers 骨干在 rectified-flow 目标下去噪两种模态，训练数据涵盖 144 万条物体级样本。
正文 · 原文
Authors:Yuhao Lu, Jiongxu Chen, Hexiang Wei, Jiajin Wu, Juelin Li, Mingjian Liang, Fan Yang, Zhuowei Liang, Zihao Liu, Zhennan Wang, Sheng Wang, Lin Shao
Abstract:Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2\% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9\% and 68.5\%, respectively, compared with its flow-only variant. Project page: this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2610.39278 [cs.CV] |
| (or arXiv:2610.39278v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39278 arXiv-issued DOI via DataCite |
Submission history
From: Yuhao Lu [view email]
[v1]
Thu, 29 Oct 2026 03:30:27 UTC (14,266 KB)
来源：arXiv cs.AI · arxiv.org