SRA：用选择性残差适配增强 LLM 推理能力
SRA: Selective Residual Adaptation for LLM Reasoning Enhancement
AI 导读
研究者提出即插即用框架 SRA（Selective Residual Adaptation），通过 Discrepance Gate 定位师生模型分布分歧位置，并用 Candidate Residual Adapter 进行 top-K logit 校准，改进蒸馏小模型的多步推理。
正文 · 原文
Abstract:Knowledge distillation transfers reasoning capabilities from large language models to lightweight students. However, distilled small models still struggle with multi-step reasoning. We observe that the distribution divergences between teacher and student models along reasoning paths are highly sparse, and target tokens almost always fall within the student top-$K$ candidate set. Based on this finding, unlike approaches updating student backbones or requiring runtime teacher computation, we propose Selective Residual Adaptation (SRA), a plug-and-play framework that internalizes localized corrections. Concretely, SRA employs a Discrepancy Gate to identify divergence positions alongside a Candidate Residual Adapter for top-$K$ logit calibration to correct the mismatch. Theoretically, we establish error bounds ensuring strict sequence level performance gains. Across mathematical benchmarks on Qwen3 models from 0.6B to 4B, SRA consistently enhances distilled student models, raising average Pass@16 by up to 4.37 percentage points while preserving general capabilities.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34199 [cs.LG] |
| (or arXiv:2610.34199v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34199 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhenxing Dou [view email]
[v1]
Sun, 25 Oct 2026 10:30:10 UTC (2,208 KB)
来源：arXiv cs.AI · arxiv.org