精度按需提升：本地小模型在不确定解码步骤逐级提高权重位宽
Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model
AI 导读
一项 arXiv 研究提出在 Qwen3-0.6B 本地解码中存储 8-bit 权重、仅用高 4-bit 解码，当输出分布熵超阈值或 top-two 间隔过小时再以 8-bit 重做该步。
正文 · 原文
Abstract:Local decoding usually fixes one weight precision for the whole generation. We study a different allocation: store 8-bit weights, decode with their top 4 bits, and redo a step at 8 bits only when the low-precision output distribution looks uncertain (entropy above a threshold or a small top-two margin). Quantization is simulated, the model is Qwen3-0.6B, and the benchmark is GSM8K. Cost is counted in weight bytes read per token. On 36724 replayed steps the entropy of the low pass ranks argmax flips with AUC 0.93. At a 15% escalation budget it catches 78% of flips and the margin gate 94%, against the 15% expected for a random gate of equal rate (a computed expectation, not a run). In 150 live problems all-high scores 64.0% and all-low 55.3% (McNemar p=0.053). An entropy gate with a 16% escalation rate scores 62.7% at 0.87 of the all-high weight bytes. Its paired difference is -1.3 (-8.0 to +5.3) points to all-high and +7.3 (+0.0 to +14.7) points to all-low (p=0.080), so the live data do not separate the gate from either end. Verified speculation reaches zero KL at similar bytes in the same cost model. A live random control, a thinking-mode phase rule, a second model and live KL were not run.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14418 [cs.AI] |
| (or arXiv:2611.14418v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14418 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Logesh L [view email]
[v1]
Tue, 10 Nov 2026 17:57:51 UTC (901 KB)
来源：arXiv cs.AI · arxiv.org