arXiv cs.AI· Tian Lyu, Yichen Jiang, Ziyuan Cheng, Yunda Yan, Xiangyu Pan, Junjie Jiang, Jialin Yu, Boyu Ning, Pietro Li\`o, Fan Mo·· 2 小时前AI 评分36
JointGV:基于并行图动力学的结构感知校正方法,提升视觉世界模型长时程预测
JointGV: Structure-Aware Correction of Visual World Models with Parallel Graph Dynamics for Long-Horizon Modeling
AI 导读
论文提出 JointGV,一种即插即用的结构感知校正方法,通过并行展开任务条件化场景图并与视觉世界模型耦合预测变化,计算有界残差来修正长时程视觉预测漂移。该方法接入 DINO-WM 和 DreamerV3,在 DMC Cheetah Run、Walker Walk 等任务上评估,32-step rollout 中将关节位置误差降低 11–28%。
正文 · 原文
Abstract:Visual world models predict future visual states conditioned on actions, but recursive errors in entity poses and spatial relations limit their long-horizon prediction for robot planning and control. We ask whether a separately predicted relational state can correct this drift without restructuring the visual model. This is challenging because graph nodes and visual states share neither elements nor a feature space. We propose Joint Graph--Visual Dynamics Correction (JointGV), a plug-in based on a cross-granularity dynamics hypothesis: despite their different representations, graph-node and visual-state changes describe the same physical transition. JointGV rolls out a task-conditioned scene graph alongside the visual world model, couples their predicted changes, and combines the resulting graph-motion context with graph-predicted node positions to compute a bounded residual. Graph-learning and correction gradients do not propagate into the original visual dynamics. We attach JointGV to DINO-WM (spatial tokens) and DreamerV3 (a global latent state), evaluating both on DMC Cheetah Run and Walker Walk across three seeds and extending the evaluation to contact-rich manipulation with DMC Finger Turn Hard. Across 32-step rollouts, JointGV improves structural and visual-prediction accuracy, reducing joint-position error by 11--28\%. For spatial tokens, learned graph--visual change coupling gives lower error than broadcasting the same mean graph-motion context to every token, particularly when non-adjacent parts appear close together in the image. On real-robot inverted-pendulum sequences, JointGV reduces DINO-WM's 32-step pole-angle error from $12.56^\circ$ to $6.69^\circ$ and improves DreamerV3 and LeWorldModel as well. In simulated inverted-pendulum control, an actor trained with JointGV-corrected rollouts completes 46 of 50 episodes, compared with none using the original rollouts.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2611.22543 [cs.RO] |
| (or arXiv:2611.22543v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22543 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tian Lyu [view email]
[v1]
Tue, 17 Nov 2026 10:33:21 UTC (5,777 KB)
来源:arXiv cs.AI · arxiv.org