arXiv cs.AI· Lin Long, Tianren Ma, Zhong Chu, Di Wu, Maosong Cao, Qixiang Ye, Junbo Zhao, Shanghang Zhang·· 3 小时前AI 评分36
Ego Guidance：用免动作人类第一视角视频增强机器人策略学习
Ego Guidance: Enhancing Robot Policy Learning with Action-Free Human Egocentric Videos
AI 导读
该研究提出 SPAR，一种基于预训练 VLM 的 MoT 架构，让人类第一视角视频在不提供动作标注的情况下增强机器人策略学习，未来视觉状态预测与动作预测并行提供独立且互补的学习信号。实验表明，从第一视角视频学到的表征可提升分布外机器人任务表现，且收益随视频数据规模扩大而增长。研究还分析了训练目标选择与数据筛选的影响。
正文 · 原文
Abstract:In this paper, we investigate how human egocentric videos can benefit robot policy learning without action supervision. Building on a pretrained VLM, we propose Shared Prediction on Action and Representation (SPAR), an MoT architecture in which future visual state prediction can provide independent and complementary learning signals alongside action prediction. We find that representations learned from egocentric videos improve downstream performance on out-of-distribution robot tasks, with further gains as video data scales. We also study the effects of objective choices and data curation, providing insights into how to leverage action-free videos for scalable robot learning.
| Comments: | Project Page: this https URL |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2611.13685 [cs.RO] |
| (or arXiv:2611.13685v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13685 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianren Ma [view email]
[v1]
Tue, 10 Nov 2026 07:33:35 UTC (2,512 KB)
来源：arXiv cs.AI · arxiv.org