扩散语言模型：从去噪动力学到输出端点不确定性的研究
From Denoising Dynamics to Endpoint Uncertainty in Diffusion Language Models
AI 导读
一项 arXiv 研究检验掩码扩散语言模型的去噪轨迹内收敛是否与最终输出的不确定性相关。研究覆盖 4 个模型、4 个推理 benchmark，共收集 32,000 条去噪轨迹、每题 20 次生成，发现轨迹内集中度越高，端点熵越低（β=-0.222，p<10⁻²⁵），且该关系随去噪进行而增强。
正文 · 原文
Abstract:Predictive uncertainty in stochastic language-model inference can be characterized by repeatedly sampling an input and measuring the distribution of outputs. This requires multiple complete generations. Masked diffusion language models expose another source of information: the evolution of the output during denoising. We ask whether convergence within diffusion trajectories is associated with uncertainty across their final outputs, or endpoints. We study four masked diffusion language models on four reasoning benchmarks, collecting 20 generations per question and 32,000 denoising trajectories in total. Within each trajectory, we measure within-run concentration as the area under the curve describing stable resolution of the eventual output. Across repeated trajectories, we measure endpoint-distribution uncertainty. Stronger within-run concentration is associated with lower endpoint entropy after accounting for model and dataset ($\beta=-0.222$, $p<10^{-25}$). This association persists for multi-token outputs and when reweighting positive-entropy cases. The relationship is asymmetric: high within-run concentration almost exclusively accompanies concentrated endpoint distributions, whereas lower concentration occurs under both concentrated and competing endpoint regimes. This relationship strengthens as denoising proceeds. Features from a single denoising run predict whether independent held-out runs will concentrate on a common endpoint or produce competing endpoints. These results connect dynamics within individual denoising runs to variability across repeated inference. Within-trajectory dynamics contain information about whether independent repetitions will concentrate on a common endpoint or produce competing endpoints, providing a route towards estimating endpoint-distribution properties with fewer complete generations.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35425 [cs.AI] |
| (or arXiv:2610.35425v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35425 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Julio Amador Diaz Lopez [view email]
[v1]
Mon, 26 Oct 2026 13:31:53 UTC (264 KB)
来源：arXiv cs.AI · arxiv.org