arXiv cs.AI· Yuehao Huang, Yunzi Wu, Xiaotao Zhang, Xinhai Li, Jiankun Dong, Sunyao Zhou, Jiajun Lv, Chi Zhang, Chenjia Bai, Yong Liu, Xuelong Li·· 5 小时前AI 评分40
WNM-3D：带 3D 场景条件化的世界导航模型，面向闭环视觉语言导航
WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN
AI 导读
研究者提出 WNM-3D，一种带 3D 场景条件化的生成式世界导航模型，用于连续视觉语言导航（VLN）。该模型用冻结的前馈几何编码器从单目 RGB 历史提取跨视图表征，经可训练的 3D Scene-to-Token Adapter 转成固定长度场景前缀，供世界动作 Diffusion Transformer 在联合去噪中通过块因果注意力持续访问。
正文 · 原文
Authors:Yuehao Huang, Yunzi Wu, Xiaotao Zhang, Xinhai Li, Jiankun Dong, Sunyao Zhou, Jiajun Lv, Chi Zhang, Chenjia Bai, Yong Liu, Xuelong Li
Abstract:Recent vision-language navigation (VLN) systems increasingly adapt pretrained vision-language models (VLMs) into vision-language-action (VLA) policies that map egocentric observations and language instructions directly to navigation actions. Although semantically capable, such action-centric training does not explicitly model how the agent's visual observations evolve under its predicted motion. Generative world-action models (WAMs) jointly predict future observations and actions, yet existing WAMs for continuous VLN do not condition joint future-view and action generation on geometry-aware representations inferred from the observation history. We present WNM-3D, a generative World Navigation Model with 3D scene conditioning for continuous VLN. A frozen feed-forward geometry encoder extracts cross-view representations from monocular RGB history, and a trainable 3D Scene-to-Token Adapter converts them into a fixed-length scene prefix for the world-action Diffusion Transformer. Through block-causal attention, this prefix remains visible to every future video-action block throughout joint denoising, providing persistent geometric context. We train WNM-3D with supervised world-action learning, DAgger-style adaptation on policy-visited states, and Counterfactual DanceGRPO refinement. Across GN-Bench and VLN-CE, WNM-3D consistently outperforms the backbone-matched WNM-2D counterpart, demonstrating gains from 3D scene conditioning across distinct visual domains. WNM-3D further outperforms strong VLN baselines on GN-Bench and demonstrates closed-loop deployment on a physical robot. Component and stage-wise ablations support the Adapter design and the complementary roles of DAgger-SFT and subsequent reward-guided refinement.
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO) |
| Cite as: | arXiv:2608.07267 [cs.AI] |
| (or arXiv:2608.07267v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.07267 arXiv-issued DOI via DataCite |
Submission history
From: Yuehao Huang [view email]
[v1]
Fri, 7 Aug 2026 14:29:00 UTC (10,688 KB)
[v2]
Thu, 20 Aug 2026 03:44:24 UTC (16,861 KB)
[v3]
Thu, 29 Oct 2026 07:13:26 UTC (18,412 KB)
来源：arXiv cs.AI · arxiv.org