RL 后训练能构建组合式推理策略
RL Post-Training Builds Compositional Reasoning Strategies
AI 导读
一项 NeurIPS 2026 海报研究在可完全观测的重写语法环境中验证:RL 后训练不只是放大基座模型的潜在能力,还能将原始技能组合成新的高级策略。仅用二值最终答案奖励的 RL 解决了预训练模型即使大幅增加采样预算也难以解出的 held-out 问题,分析显示 RL 先强化原始归约、再发现顺序组合与并行组合等可复用的组合程序;与拒绝采样微调相比,关键差异在于选择性而非探索量。
正文 · 原文
Abstract:Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.
| Comments: | Accepted at NeurIPS 2026 (poster) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.07646 [cs.AI] |
| (or arXiv:2607.07646v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07646 arXiv-issued DOI via DataCite |
Submission history
From: Nishil Patel [view email]
[v1]
Wed, 8 Jul 2026 17:04:42 UTC (1,054 KB)
[v2]
Tue, 17 Nov 2026 17:20:07 UTC (1,042 KB)
来源:arXiv cs.AI · arxiv.org