Echo：用离策略经验扩展视觉强化学习
Echo: Scaling Visual Reinforcement Learning with Off-policy Experience
AI 导读
Echo 是一个单阶段离策略强化学习框架，通过逆向蒸馏让基于状态的辅助策略模仿视觉策略并收集无需渲染的廉价经验，从而在不等比增加视觉计算的前提下扩展经验采集。该框架基于 FastTD3 实现，在 Genesis 中评测：相比 SDPG，三项运动控制任务达标时间平均缩短 60%，峰值平均回报提升 62%；两项灵巧手任务分别缩短 56% 和提升 12%。
正文 · 原文
Abstract:Scaling visual reinforcement learning (RL) with GPU-accelerated simulation is computationally expensive due to rendering and visual-policy inference costs. Consequently, two-stage teacher--student distillation is commonly used to train visual policies, but introduces an imitation gap. In this preliminary study, we introduce \echomethod{}, \textbf{a single-stage off-policy RL framework that scales experience collection without proportionally increasing visual computation}. Through \textbf{inverse distillation}, a state-based auxiliary policy imitates the evolving visual policy and collects inexpensive, unrendered experience. Echo leverages this experience for off-policy value learning, directly improving the visual policy through a shared privileged critic. We instantiate \echomethod{} with FastTD3 and evaluate it on three visual control tasks and two dexterous manipulation tasks in Genesis. Relative to \sdpgmethod{} under the evaluated configurations, Echo reduces time to reference returns by an average of \textbf{60\% across the three locomotion tasks} and \textbf{56\% across both hand tasks}, while improving peak mean returns by \textbf{62\% and 12\%}, respectively. Both hand tasks also achieve higher mean returns at matched simulator-transition budgets. These initial results suggest that scaling off-policy experience is a promising approach to efficient direct visual RL.
| Comments: | Report. 16 pages |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.37865 [cs.RO] |
| (or arXiv:2610.37865v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37865 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Gagan Khandate [view email]
[v1]
Tue, 27 Oct 2026 19:11:05 UTC (774 KB)
来源：arXiv cs.AI · arxiv.org