arXiv cs.AI· Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn, Krishnan Srinivasan, Animesh Garg, Vitor Campagnolo Guizilini, Paarth Shah·· 4 小时前AI 评分53
TAPESTRY 论文：用单一 Transformer 骨干同时支持视频生成与机器人控制
TAPESTRY: A Single Backbone for Video Generation and Robot Control
AI 导读
TAPESTRY 论文提出把机器人动作作为视频 patch token 注入预训练视频扩散 Transformer，形成无需推理时视频生成的世界动作模型。通过视频扩散、视频+动作联合扩散和动作-only 扩散三个目标共享骨干优化，测试时单步 DDIM 生成动作序列，每个动作块约 84 ms，在仿真与真机实验中性能强且推理成本明显低于同类联合视频-动作策略，已被 CoRL 2026 接收。
正文 · 原文
Abstract:Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at this https URL.
| Comments: | Accepted to Conference on Robot Learning (CoRL) 2026 |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.36823 [cs.RO] |
| (or arXiv:2610.36823v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36823 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jeremy Collins [view email]
[v1]
Tue, 27 Oct 2026 08:23:30 UTC (3,146 KB)
来源：arXiv cs.AI · arxiv.org