FloMo：将 3D 场景流作为世界动作模型中间表示，从人类视频学习机器人动作
FloMo: 3D Scene Flow as a World Action Model Intermediate for Learning from Human Video
AI 导读
arXiv 论文提出 FloMo，一个经微调用于预测 3D scene-flow 和动作 token 的 5B 视频生成模型，主张密集 3D 运动是视频预训练向机器人动作迁移的最佳媒介。
正文 · 原文
Abstract:Pretrained video generation models distill internet-scale motion priors into robot policies, but full video is a redundant signal for action prediction. We argue that dense 3D motion is the medium through which video pretraining best transfers to robot actions, and that predicting motion rather than video affords better sample-efficiency for action prediction. FloMo is a 5B video-generation model fine-tuned to predict 3D scene-flow and action tokens. Rendering scene flow as RGB and encoding it into the same latent space as video enables a pretrained video generation backbone to efficiently adapt to predict 3D scene flow. Co-trained on action-free human video and action-labeled robot teleoperation data, our model benefits from human-video pretraining and generalizes to unseen objects, motion primitives, and tasks. Replacing motion prediction with a video prediction objective degrades out-of-distribution generalization, while adding a video prediction target alongside motion lowers policy success rather than improving it. When motion, video, and action are predicted jointly, action tokens attend far more to motion than to video, suggesting that predicting video wastes model capacity. Together, these results indicate that motion effectively mediates the transfer from video pretraining to action. More information can be found at this http URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.05062 [cs.CV] |
| (or arXiv:2611.05062v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05062 arXiv-issued DOI via DataCite |
Submission history
From: Shitij Govil [view email]
[v1]
Tue, 3 Nov 2026 15:25:35 UTC (725 KB)
[v2]
Wed, 4 Nov 2026 03:57:24 UTC (725 KB)
来源：arXiv cs.AI · arxiv.org