通过长短对比剪枝提升剪枝后推理模型的思维链可压缩性
Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning
AI 导读
论文提出对比集合差分剪枝方法（CSD），通过对比同一问题长短正确答案的神经元激活，识别并优先剪除冗余推理神经元，并扩展偏好对到剪枝模型总能或总不能解决的问题。在 DeepSeek-R1-Distill-Qwen-7B 上剪枝 20% 后，相比推理感知校准（RAC）加压缩的组合，在四个数学 benchmark 上少用 8–22% 思考 token，准确率差距在 0.5 分以内。
正文 · 原文
Abstract:Large reasoning models are costly to serve because they are large and because they generate long chains of thought (CoT). Weight pruning and CoT compression address these two costs separately, but they interfere in that pruning adds redundant reasoning on problems the model still solves and incomplete reasoning on those it no longer solves, and on harder problems makes the model less compressible by subsequent length-preference optimization. We trace this to a blind spot on each side: pruning criteria do not distinguish necessary from redundant reasoning, and preference-based CoT compression receives no training signal on problems a pruned model always or never solves. We propose Contrastive Set-Difference (CSD), which identifies neurons enriched in redundant reasoning by contrasting their activations on long and short correct solutions of the same problem and prunes them preferentially, and we extend preference pairs to the problems a pruned model always or never solves. Pruning 20\% of DeepSeek-R1-Distill-Qwen-7B, the combination uses 8--22\% fewer thinking tokens on four math benchmarks than pruning with reasoning-aware calibration (RAC) followed by the same compression, with accuracy within 0.5 points.
| Comments: | 11 pages. Preliminary version; further experiments and revisions are planned |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34336 [cs.CL] |
| (or arXiv:2610.34336v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34336 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tao Jing [view email]
[v1]
Sun, 25 Oct 2026 14:52:21 UTC (38 KB)
来源：arXiv cs.AI · arxiv.org