arXiv cs.AI· Jameson Sandler, Shane Bergsma, Mostafa Elhoushi, Bin Claire Zhang, Gavia Gray, Joel Hestness, Nolan Dey, Ferdinando Fioretto·· 56 分钟前AI 评分35
DNeo:并行草稿模型需要更好的目标特征编码
DNeo: Your Parallel Draft Model Needs Better Target Feature Encoding
AI 导读
DNeo 提出 Mixture-of-Features Attention(MoFA)特征编码方法,针对标准特征编码有损(residual mixing)、无记忆、均匀采样目标层三个局限,引入拒绝记忆、2-tap 因果卷积、prefix-only 交叉注意力和非均匀层选择。
正文 · 原文
Abstract:Speculative-diffusion decoding uses a small parallel drafter model to generate b draft tokens in one forward pass, which then undergo parallel verification via a larger target model. Faster inference can be achieved when multiple draft tokens pass verification simultaneously. Recent parallel draft architectures (DFlash, DSpark) improve draft quality by conditioning on rich features computed by the target, through a mechanism known as feature encoding. However, standard feature encoding mechanisms have three limitations: (1) they are lossy, applying encoding projections that are destructive to the underlying target feature-space via a discovered effect known as residual mixing, (2) they are memory-less, with no record of previously processed proposals, and (3) they utilize features sampled uniformly from across the target layers, omitting important hidden states. To address these issues, we propose Mixture-of-Features Attention (MoFA), a lightweight feature-encoding approach equipped with a working memory that encodes previously rejected proposals. Leveraging MoFA, rejection memory, and a set of further enhancements, including 2-tap causal convolutions, prefix-only cross-attention, and non-uniform target layer selection, the final architecture, DNeo, achieves 1.3x faster generation speed than SoTA (DFlash-2) without impacting drafter footprint or generation quality. DNeo marks a new state of the art in high quality, fast LLM inference.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.21925 [cs.AI] |
| (or arXiv:2611.21925v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21925 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jameson Sandler [view email]
[v1]
Mon, 16 Nov 2026 21:41:20 UTC (1,903 KB)
来源:arXiv cs.AI · arxiv.org