闭环课程自适应何时帮助 GRPO？基于 Countdown 的分析
When Does Closed-Loop Curriculum Adaptation Help GRPO? An Analysis on Countdown
AI 导读
一项研究提出轻量级自适应课程方法 CLOSEDLOOPGAUSSIAN，用于提升大语言模型强化微调效率。该方法保持由易到难 Gaussian 采样结构，通过周期性探测各难度准确率，动态调节训练进度与采样范围，且不改变奖励函数、模型架构或 GRPO 优化器。在符号算术推理任务上，其精确匹配准确率超过原生 GRPO 和固定开环课程，在中等难度区间收益最大，代码已开源。
正文 · 原文
Abstract:Reinforcement fine-tuning has become an effective post-training paradigm for improving the reasoning ability of large language models, but it remains sample-inefficient when the training distribution is mismatched to the model's current capability. In group-relative objectives such as Group Relative Policy Optimization (GRPO), problems that are too hard often produce uniformly failed rollouts, while problems that are too easy provide little useful advantage signal. In this work, we introduce CLOSEDLOOPGAUSSIAN, a lightweight adaptive curriculum method for efficient reinforcement fine-tuning on ordered reasoning tasks. Instead of following a fixed easy-to-hard schedule, CLOSEDLOOPGAUSSIAN preserves the structure of an Easy-to-Hard Gaussian sampler while dynamically adjusting its pace and spread using periodic probe estimates of per-difficulty accuracy. The controller accelerates training when the model is ahead of a reference curriculum, slows down when the model falls behind, and broadens sampling when earlier difficulties regress. This feedback mechanism keeps training focused on tasks that are challenging but still learnable, without changing the reward function, model architecture, or underlying GRPO optimizer. Experiments on symbolic arithmetic reasoning task show that CLOSEDLOOPGAUSSIAN improves exact-match accuracy over vanilla GRPO and fixed open-loop curricula, with the strongest gains in intermediate-difficulty regimes where group-relative learning signals are most informative. Codes are available at this https URL.
| Comments: | 6 pages, 2 figures, Accepted at IEEE ICTAI 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39056 [cs.LG] |
| (or arXiv:2610.39056v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39056 arXiv-issued DOI via DataCite |
Submission history
From: Yuv Raj Pant [view email]
[v1]
Wed, 28 Oct 2026 23:35:56 UTC (4,772 KB)
来源：arXiv cs.AI · arxiv.org