arXiv cs.AI· Yukai Jin, Zhankai Ye, Jiayue Yang, Fangyi Li, Shuoqiu Li, Xu Gao, Shibo Li, Xin Liu, Shangqian Gao·· 3 小时前AI 评分35
TraceQuery：用全去噪轨迹指导扩散语言模型的高效稀疏解码
TraceQuery: learning what to recompute from full-denoising trajectory in diffusion language models
AI 导读
一项 arXiv 研究提出 TraceQuery，一种面向扩散大语言模型（dLLM）的轻量级稀疏解码框架。该方法基于“缓存生成在贴近全去噪轨迹时保持准确”的发现，用轨迹监督预测少量近期与轨迹相关的位置，仅对这些候选位置做新计算。TraceQuery 在缓存类方法中取得最佳的质量-计算权衡，大幅减少新增 token 计算量，并可跨任务、模型变体和模型族泛化。
正文 · 原文
Abstract:Diffusion large language models (dLLMs) enable flexible generation through iterative masked denoising, but incur substantial inference cost because token representations are repeatedly recomputed across denoising steps. Existing cache-based methods reduce this cost by reusing intermediate results across denoising steps and recomputing only a subset of positions at each step, chosen either on a fixed schedule or by heuristic signals of change. Our trajectory experiments instead point to decoding trajectory as the more relevant quantity. Cached generation remains accurate when it stays close to the full-denoising trajectory, with errors concentrated around unmask-order deviations and reduced as longer reference prefixes are recovered. This suggests a more direct principle: limiting representation error helps, but keeping generation on the full-denoising trajectory is a more direct route to accurate decoding. We therefore formulate efficient dLLM inference as trajectory-guided querying. Rather than broadly repairing representation error, we use full-denoising trajectories to supervise which unresolved positions should receive limited fresh computation. Based on this formulation, we introduce TRACEQUERY, a lightweight sparse decoding framework that predicts a small set of near-future trajectory-relevant positions, queries only those candidates, and lets the backbone select which one to commit from their fresh predictions. TRACEQUERY delivers the strongest quality-computation trade-off among cache-based approaches, sustaining high accuracy with substantially fewer fresh token computations, and generalizes across tasks, model variants, and families. Code: this https URL
| Comments: | 9 pages of main body, 5 pages of appendix, 4 figures, 1 table |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33008 [cs.CL] |
| (or arXiv:2610.33008v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33008 arXiv-issued DOI via DataCite |
Submission history
From: Yukai Jin [view email]
[v1]
Wed, 7 Oct 2026 18:20:13 UTC (1,671 KB)
来源：arXiv cs.AI · arxiv.org