DRAMA：剖析注意力冗余以加速多模态扩散大语言模型
DRAMA: Dissecting Attention Redundancy for Accelerating Multimodal Diffusion Large Language Models
AI 导读
DRAMA 是一个无需训练的多模态 dLLM 推理加速框架，通过剖析注意力张量发现视觉 token 稀疏集中和注意力分布早期稳定两类冗余结构，并提出 SVTP 视觉 token 剪枝与 SGAS 自适应停止两项机制。在六个视觉语言 benchmark 上平均实现 18.67 倍推理加速，无需训练、架构修改或注意力缓存，精度保持有竞争力。该论文已被 NeurIPS 2026 接收。
正文 · 原文
Abstract:Multimodal diffusion large language models (dLLMs) generate text through iterative masked denoising with bidirectional attention, offering a compelling alternative to autoregressive vision-language modeling. However, each denoising step requires a full forward pass over long multimodal sequences in which visual tokens dominate the sequence length, and a fixed step budget further compounds this quadratic per-step cost. Existing inference acceleration approaches for multimodal dLLMs characterize attention redundancy implicitly, motivating designs from isolated attention visualizations rather than from measurement. We close this gap with DRAMA, a training-free acceleration framework grounded in a systematic dissection of attention redundancy in multimodal dLLMs. By probing attention tensors along spatial and temporal axes, we reveal two persistent redundancy structures: (i) attention mass consistently concentrates on a sparse subset of visual tokens, and (ii) attention distributions rapidly stabilize across denoising steps after an initial formation phase. These findings motivate two complementary mechanisms: Spatial-Redundancy-Aware Visual Token Pruning (SVTP) and Sufficiency-Guided Adaptive Stopping (SGAS). Together, DRAMA achieves an average 18.67 times inference speedup across six vision-language benchmarks without training, architectural modification, or attention caching, while maintaining competitive accuracy. Our results establish a strong, measurement-grounded acceleration baseline for multimodal dLLM inference.
| Comments: | Accepted to NeurIPS 2026. 27 pages, 19 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.16345 [cs.LG] |
| (or arXiv:2611.16345v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16345 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yi-Hsin Hung [view email]
[v1]
Thu, 12 Nov 2026 06:04:08 UTC (2,315 KB)
来源：arXiv cs.AI · arxiv.org