arXiv cs.AI· Carel van Niekerk, Renato Vukovic, Benjamin Ruppik, Hsien-chin Lin, Shutong Feng, Milica Ga\v{s}i\'c·· 4 小时前AI 评分40
基于自我反馈强化学习（RLSF）的后训练方法提升大语言模型校准与推理
Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
AI 导读
研究者提出自我反馈强化学习 RLSF，一种利用模型内在置信度作为自生成奖励的后训练方法。该方法从冻结的 LLM 生成多条 CoT 解码路径，按答案置信度排序构建合成偏好，再通过标准偏好优化微调，无需人工标注、标准答案或外部奖励。RLSF 同时改善概率校准和逐步推理，在算术推理与选择题问答上取得提升。
正文 · 原文
Abstract:Large Language Models (LLMs) often produce plausible but poorly-calibrated answers, limiting their reliability on reasoning-intensive tasks. Recent research suggests that Chain-of-Thought (CoT) reasoning paths are inherent in pre-trained LLMs and can be elicited by simply altering the decoding process, where the presence of a CoT path correlates with higher answer confidence. Building on these insights, we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward. By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences. These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards. RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering. By converting a model's own uncertainty into structured self-feedback, RLSF affirms reinforcement learning on intrinsic model behaviour as a principled and data-efficient component of the LLM post-training pipeline. Our results demonstrate that leveraging these inherent reasoning capabilities provides a robust path for enhancing model reliability without manual prompt engineering or external supervision.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2507.21931 [cs.CL] |
| (or arXiv:2507.21931v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2507.21931 arXiv-issued DOI via DataCite |
Submission history
From: Renato Vukovic [view email]
[v1]
Tue, 29 Jul 2025 15:46:26 UTC (229 KB)
[v2]
Tue, 15 Sep 2026 11:56:06 UTC (775 KB)
[v3]
Thu, 5 Nov 2026 09:31:33 UTC (279 KB)
来源：arXiv cs.AI · arxiv.org