arXiv cs.AI· Yuan Zhuang, Yuexin Bian, Sihong He, Jie Feng, Qing Su, Songyang Han, Jonathan Petit, Shihao Ji, Yuanyuan Shi, Fei Miao·· 5 小时前AI 评分38
LoRA 用于 Off-Policy 强化学习 Critic 学习的低秩正则化研究
Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning
AI 导读
一项 NeurIPS 2026 论文提出将 LoRA 作为 off-policy 强化学习中 critic 学习的结构化正则化手段：冻结随机初始化的基础矩阵，仅优化低秩适配器。
正文 · 原文
Abstract:Scaling critic capacity is a promising direction for improving off-policy reinforcement learning (RL). However, recent work shows that larger critics are prone to overfitting and instability in replay-based bootstrapped training. In this paper, we propose using Low-Rank Adaptation (LoRA) as a structural regularizer for critic learning from scratch. Our approach freezes randomly initialized base matrices and optimizes only the corresponding low-rank adapters, thereby constraining the learned weight changes to low-rank residuals. We evaluate our method on state-based continuous-control tasks, using SAC with SimbaV2 and BRC on DMC-Hard, and FastTD3 with its SimbaV2 variant on IsaacLab. Empirically, LoRA reduces critic loss during training and improves overall policy performance, achieving the best or competitive results on most tasks. Extensive experiments demonstrate that our low-rank updates provide a simple and effective form of structural regularization for critic learning in off-policy RL.
| Comments: | 2026 Conference on Neural Information Processing Systems (NeurIPS 2026) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.18978 [cs.LG] |
| (or arXiv:2604.18978v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.18978 arXiv-issued DOI via DataCite |
Submission history
From: Yuan Zhuang [view email]
[v1]
Tue, 21 Apr 2026 01:59:54 UTC (4,126 KB)
[v2]
Thu, 7 May 2026 15:47:31 UTC (2,855 KB)
[v3]
Thu, 29 Oct 2026 16:01:54 UTC (2,888 KB)
来源：arXiv cs.AI · arxiv.org