arXiv cs.AI· Jialin Yang, Jiankun Wang, Jiajun Wu, Henry Leung, Jiayu Zhou, Steve Drew·· 4 小时前AI 评分40
D-RPC：通过推理空间压缩实现结构化推理蒸馏
Structural Rationale Distillation via Reasoning Space Compression
AI 导读
研究者提出 D-RPC（通过推理路径压缩的蒸馏）方法，让 LLM 教师模型遵循一个动态维护的可复用高层推理路径库，为相似问题生成结构一致的推理依据，再蒸馏给小模型。PAC-Bayes 分析刻画了路径库规模与覆盖面的权衡。在五个数学与常识推理基准、两个学生模型上，D-RPC 在评测的蒸馏方法中取得最高平均准确率，且消耗 token 少于模板密集型方法。
正文 · 原文
Abstract:When distilling reasoning from large language models (LLMs) into smaller ones, teacher rationales for similar problems often vary wildly in structure and strategy. Like a chef who makes the same dish differently each time, this inconsistency burdens the student with noisy supervision that is hard to internalize. We propose Distillation through Reasoning Path Compression (D-RPC), which constrains the teacher to follow a compact, dynamically maintained bank of reusable high-level reasoning paths. For each training question, D-RPC retrieves the most relevant path and conditions the teacher to follow it, producing rationales that are consistent across similar problems yet diverse enough to cover different problem types. A PAC-Bayes analysis formalizes a trade-off between bank size and coverage, with smaller banks limiting path diversity at the risk of coverage gaps. Our ablations favor an intermediate bank size, consistent with this trade-off. Across five math and commonsense reasoning benchmarks with two student models, D-RPC achieves the highest average accuracy among the evaluated distillation methods, while using fewer tokens than template-heavy alternatives.
| Comments: | NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.07139 [cs.CL] |
| (or arXiv:2605.07139v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07139 arXiv-issued DOI via DataCite |
Submission history
From: Jialin Yang [view email]
[v1]
Fri, 8 May 2026 02:15:52 UTC (549 KB)
[v2]
Wed, 4 Nov 2026 02:18:35 UTC (546 KB)
来源：arXiv cs.AI · arxiv.org