arXiv cs.AI· Haoran Yuan, Weigang Yi, Zhenyu Zhang, Wendi Chen, Yuchen Mo, Jiashi Yin, Xinzhuo Li, Yue Luo, Jialin Li, Xiangyu Zeng, Chuan Wen, Cewu Lu, Katherine Driggs-Campbell, Ismini Lourentzou·· 4 小时前AI 评分46
VTAM：无需触觉预训练的高效视觉-触觉世界-动作模型
VTAM: Efficient Visuo-Tactile World-Action Models for Contact-Rich Manipulation without Tactile Pretraining
AI 导读
研究者提出 VTAM，一种无需触觉或动作预训练的视觉-触觉世界-动作模型，用于接触密集型操作。它复用冻结的 video VAE 与预训练视频 Transformer 权重，在 4 个真实机器人任务上超越所有基线，在 8 个 UniVTAC 仿真任务上平均成功率超过所有无触觉预训练基线，并以 83.4% 对 84.5% 接近基于超 30K 小时数据预训练的 N0-TWAM。
正文 · 原文
Authors:Haoran Yuan, Weigang Yi, Zhenyu Zhang, Wendi Chen, Yuchen Mo, Jiashi Yin, Xinzhuo Li, Yue Luo, Jialin Li, Xiangyu Zeng, Chuan Wen, Cewu Lu, Katherine Driggs-Campbell, Ismini Lourentzou
Abstract:Tactile vision-language-action models (VLAs) and tactile world-action models (WAMs) are a promising new paradigm for contact-rich manipulation, but existing models either underutilize touch, rely on large-scale tactile pretraining, or incur high inference latency due to iterative video generation. We present VTAM, an efficient visuo-tactile world-action model for contact-rich manipulation that requires no tactile or action pretraining. RGB and tactile observations share a frozen video VAE and the pretrained video transformer weights. VTAM first adapts the video backbone to jointly predict future visual and tactile latents, then fine-tunes it end-to-end with a randomly initialized action expert that predicts actions, proprioceptive states, and a light flow-force proxy computed from tactile optical flow. Trained only on task demonstrations, VTAM outperforms all baselines on four real-robot tasks across two platforms with different tactile sensors, outperforms all baselines without tactile pretraining in average success on eight UniVTAC simulation tasks, and is comparable to tactile VLAs and WAMs pretrained on over 30K hours of largely visuo-tactile robot data (83.4% vs. 84.5% for N0-TWAM). Without full video denoising at inference, VTAM is approximately 6x and 16x faster than N0-TWAM and LingBot-VA without compilation, with compiled latency comparable to Fast-WAM. Ablations show that feeding frozen-VAE tactile features directly to the action expert, bypassing tactile world modeling, fails on every tested task, that Stage I visuo-tactile world-model fine-tuning improves UniVTAC success on most tasks, and that flow-force proxy keeps the policy from fully ignoring touch. The tactile encoder transfers zero-shot across GelSight Mini and Xense Photon sensors.
| Comments: | 12 pages. Project page: this https URL |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2603.23481 [cs.RO] |
| (or arXiv:2603.23481v2 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2603.23481 arXiv-issued DOI via DataCite |
Submission history
From: Haoran Yuan [view email]
[v1]
Tue, 24 Mar 2026 17:45:06 UTC (36,714 KB)
[v2]
Thu, 5 Nov 2026 07:33:10 UTC (18,191 KB)
来源：arXiv cs.AI · arxiv.org