Semantic World Models：将视觉语言模型训练为语义世界模型用于机器人规划
Semantic World Models
AI 导读
一项 arXiv 研究提出"语义世界模型"方法：不做未来像素重建，而是把世界建模转化为对未来帧语义信息的视觉问答任务，通过图像-动作-文本数据的监督微调将视觉语言模型训练为世界模型。该方法在仿真机器人任务上验证，性能优于典型的基于重建的动作条件世界建模范式，并继承了预训练视觉语言模型的泛化与鲁棒性。
正文 · 原文
Abstract:Planning with world models offers a powerful paradigm for robotic control. Conventional approaches train a model to predict future frames conditioned on current frames and actions, which can then be used for planning. However, the objective of predicting future pixels is often at odds with the actual planning objective; strong pixel reconstruction does not always correlate with good planning decisions. This paper posits that instead of reconstructing future frames as pixels, world models only need to predict task-relevant semantic information about the future. For such prediction the paper poses world modeling as a visual question answering problem about semantic information in future frames. This perspective allows world modeling to be approached with the same tools underlying vision language models. Thus, vision-language models can be trained as "semantic" world models through a supervised finetuning process on image-action-text data, enabling planning for decision-making while inheriting many of the generalization and robustness properties from the pretrained vision-language models. In this paper, we introduce this new method and validate it on simulated robotic tasks leading to improvements over typical paradigms of reconstruction-based action-conditional world modeling. Website available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2510.19818 [cs.LG] |
| (or arXiv:2510.19818v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.19818 arXiv-issued DOI via DataCite |
Submission history
From: Chuning Zhu [view email]
[v1]
Wed, 22 Oct 2025 17:53:45 UTC (2,153 KB)
[v2]
Mon, 26 Oct 2026 20:48:32 UTC (2,150 KB)
来源：arXiv cs.AI · arxiv.org