ARCHER:让小语言模型从模仿推理文本走向可执行推理的两阶段训练框架
ARCHER: From Rationale Imitation to Executable Reasoning in Small Language Models
AI 导读
ARCHER 是一个两阶段训练框架,先把自由形式的思维链编译为包含推理步骤、计算器调用与验证动作的类型化动作轨迹,再通过非对称分层强化学习优化该策略。
正文 · 原文
Abstract:Small language models are attractive for local, low-cost, and privacy-sensitive deployment, but they remain unreliable on mathematical reasoning tasks that require decomposition, exact computation, tool use, and verification. We introduce \textbf{ARCHER}: \textbf{A}ction-trace \textbf{R}einforcement for \textbf{C}alculator-grounded \textbf{H}ierarchical \textbf{E}xecutable \textbf{R}easoning, a two-stage framework for training small language models (SLMs) to produce executable and auditable reasoning trajectories. In the first stage, ARCHER compiles free-form chain-of-thought rationales into typed action traces containing reasoning steps, calculator calls, deterministic runtime observations, verification actions, and final answers. This shifts the supervision target from imitating explanatory text to learning a tool-grounded reasoning policy. In the second stage, ARCHER refines this policy through asymmetric hierarchical reinforcement learning, using grouped and conditionally gated trajectory rewards for answer correctness, tool grounding, process discipline, and trace alignment. Across five standard sub-10B backbones, ARCHER improves over the strongest tool-augmented baseline by 5.51 percentage points on macro-average across benchmark accuracies, including gains of 8.95 points on MATH-500 and 5.33 and 5.34 points on AIME-2024 and AIME-2025. Evaluations on two additional reasoning-specialized 7B backbones further show that executable grounding remains beneficial after reasoning-focused reinforcement learning or distillation. These results demonstrate that executable supervision and reward-guided behavioral refinement provide a general and effective training paradigm for compact mathematical reasoners.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.18494 [cs.AI] |
| (or arXiv:2611.18494v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.18494 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dhruv Dixit [view email]
[v1]
Fri, 13 Nov 2026 18:05:33 UTC (10,629 KB)
来源:arXiv cs.AI · arxiv.org