arXiv cs.AI· Zijian Jia, Peixuan Zhang, Genying Li, Di Wu, Yujie Fu, Si Li, Boxin Shi, Shuchen Weng·· 2 小时前AI 评分34
LLaCO：面向大语言模型的连续扩散与竞争感知语义调度
LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling
AI 导读
论文提出 LLaCO，一个基于竞争感知语义调度的连续扩散文本生成框架，并提出竞争阈值理论（CTT）与语义保留调度（SRS）来对齐去噪轨迹与身份衰减进程。该框架通过嵌入方向适配器与时间步条件调制适配预训练骨干，采用球面加噪过程消除范数衰减。8B 版本的 LLaCO 超越此前连续扩散基线，媲美同规模离散扩散模型。
正文 · 原文
Abstract:Continuous diffusion models have emerged as a promising non-autoregressive paradigm for text generation over continuous embedding spaces. Along the continuous trajectory, the residual token identity directly shapes the denoising objective. However, the geometric trajectory misaligns with temporal progression, triggering severe temporal compression. In this paper, we show that embedding norm decay causes Euclidean distance to yield spurious overtaking, indicating that token competition must be evaluated directionally. In this space, a candidate overtakes the source token only when the effective noise magnitude surpasses their semantic separation. We formalize these pairwise dynamics as the Competition Threshold Theory (CTT). Crucially, high-dimensional overtaking requires extreme noise levels, causing standard trajectories to concentrate identity decay into a narrow window near the noisy endpoint. To resolve this, we propose Semantic Retention Scheduling (SRS) to align time allocation with the progression of identity decay, and demonstrate that existing token identity metrics are fundamentally statistical aggregations of these pairwise crossings. Building on these theoretical insights, we introduce LLaCO, a generative framework powered by competition-aware semantic scheduling. LLaCO adapts a pretrained backbone to continuous representations via an embedding directional adapter and timestep-conditioned modulation. To operate within the directional space, LLaCO employs a spherical noising process formulated as a geodesic interpolation toward a mask-biased endpoint, eliminating norm decay while preserving accurate token competition. Guided by SRS, LLaCO reallocates the denoising trajectory proportionally to the logarithmic rank decay. Instantiated as an 8B model, LLaCO outperforms previous continuous diffusion baselines and rivals discrete diffusion models of comparable scale.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32604 [cs.AI] |
| (or arXiv:2610.32604v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32604 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zijian Jia [view email]
[v1]
Fri, 23 Oct 2026 13:28:40 UTC (463 KB)
来源：arXiv cs.AI · arxiv.org