arXiv cs.AI· Sami Nur Islam, Johan Obando-Ceron, Doina Precup, Aaron Courville, Pablo Samuel Castro·· 4 小时前AI 评分37
冻结而非克隆：无离线数据保留下的强化学习微调研究
Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data
AI 导读
一项研究指出，离线转在线强化学习中离线数据不可用时，失败原因除 critic 侧问题外，还存在 actor 侧失败：继续在线行为克隆（BC）会抹除预训练先验，引发表征坍缩和动作多样性丧失，使 critic 缺乏有用的探索。
正文 · 原文
Abstract:Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.36247 [cs.LG] |
| (or arXiv:2610.36247v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36247 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Johan Obando-Ceron [view email]
[v1]
Mon, 26 Oct 2026 20:28:50 UTC (1,693 KB)
来源：arXiv cs.AI · arxiv.org