LGPO：大语言模型强化学习中重要性比率是否必要
Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?
AI 导读
一项发表于 NeurIPS 2026 的研究提出 Likelihood-Gated Policy Optimization（LGPO），用基于 likelihood 的门控实现软信任域约束，无需计算旧策略 likelihood。
正文 · 原文
Abstract:Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios. In this work, we propose Likelihood-Gated Policy Optimization (LGPO), which enforces a soft trust region constraint via likelihood-based gating, eliminating the need to compute old-policy likelihoods. Empirically, we show that removing the importance sampling correction term does not harm training stability, whereas removing the trust region mechanism can lead to collapse. Moreover, ratio-based clipping can fail in fully on-policy training: the importance ratio stays at 1, so the ratio-based trust region constraint never activates. Under standard training settings where GRPO is stable, LGPO achieves comparable training stability and peak performance while reducing training time by ~18% on average relative to a standard implementation that recomputes old-policy likelihoods. In fully on-policy training, where GRPO fails, LGPO remains stable, enabling more efficient and robust LLM RL post-training across training regimes.
| Comments: | 21 pages, 11 figures, 5 tables. Accepted at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.15404 [cs.LG] |
| (or arXiv:2611.15404v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.15404 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junhyuck Kim [view email]
[v1]
Wed, 11 Nov 2026 12:31:55 UTC (502 KB)
来源：arXiv cs.AI · arxiv.org