NVIDIA OmniDreams 发布实时生成式世界模型用于闭环自动驾驶仿真
NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation
NVIDIA 团队提出 OmniDreams，一个从 Cosmos diffusion 模型中训练和后训练的生成式世界模型，可实时自回归生成动作条件视频，并利用 21k 小时驾驶场景数据合成极端天气和不可预测动态行为等传统模拟器难以捕捉的现象。
原文给出用生成式世界模型做闭环自动驾驶仿真和作为策略骨干的初步结果，读者可了解其训练规模与参数效率。
Authors:Aarti Basant, Amlan Kar, Despoina Paschalidou, Fangyin Wei, Francesco Ferroni, Guillermo Garcia Cobo, Haithem Turki, Huan Ling, Jaewoo Seo, James Lucas, Jay Zhangjie Wu, Jialiang Wang, Jonathan Lorraine, Jun Gao, Kai He, Katarina Tothova, Kevin Xie, Michal Tyszkiewicz, Qi Wu, Riccardo de Lutio, Ruilong Li, Sanja Fidler, Seung Wook Kim, Tianchang Shen, Tianshi Cao, Tobias Pfaff, William Lew, Xindi Wu, Xuanchi Ren, Yifan Lu, Yuxuan Zhang, Zan Gojcic, Zian Wang
Abstract:As autonomous vehicle capabilities advance, the safe evaluation of driving policies in long-tail scenarios remains a critical bottleneck. In closed-loop simulation, the driving policy model actively interacts with the environment, where its actions dynamically update the simulator state and directly influence the next set of generated sensor observations. While recent reconstruction-based neural simulators offer photorealism, they are fundamentally constrained by their initial captured data and struggle to generalize to highly dynamic or novel scenes. To overcome these limitations, we introduce OmniDreams, a foundation generative world model mid- and post-trained from the Cosmos diffusion model to autoregressively generate action-conditioned videos in real time. By leveraging the rich visual priors of Cosmos and mid- and post-training on 21k hours of driving scenarios, OmniDreams synthesizes complex, unobserved phenomena that are hard for traditional simulators to capture, such as extreme weather and unpredictable dynamic agent behaviors. Crucially, it autoregressively conditions its photorealistic sensor generation on past frames, the current simulator state, and immediate driving actions. Deployed in a closed-loop system with the Alpamayo 1 policy model and AlpaSim orchestrator, OmniDreams acts as a highly responsive, reactive environment, providing a scalable and comprehensive solution for training and evaluating next-generation autonomous driving policies. We additionally show preliminary results indicating that a world-action model (WAM) post-trained from OmniDreams achieves strong performance on the Physical AI Autonomous Vehicles NuRec dataset, surpassing the VLA-based Alpamayo 1.5 research policy model while using only 1/5 the total parameters. These results highlight the potential for a real-time world model like OmniDreams to also serve as a backbone for policy architectures.
| Comments: | Research blog: this https URL, GitHub: this https URL, Model weights: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2606.03159 [cs.CV] |
| (or arXiv:2606.03159v4 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.03159 arXiv-issued DOI via DataCite |
Submission history
From: Jialiang Wang [view email]
[v1]
Tue, 2 Jun 2026 05:11:05 UTC (45,358 KB)
[v2]
Thu, 23 Jul 2026 22:35:14 UTC (22,680 KB)
[v3]
Wed, 23 Sep 2026 19:11:35 UTC (22,679 KB)
[v4]
Thu, 12 Nov 2026 05:36:26 UTC (22,680 KB)
来源：arXiv cs.AI · arxiv.org