arXiv cs.AI· Tianqi Chen, Xunchao Li, Linchuan Xie, Guanghua Yu, Jianchen Zhu, Peisong Wang, Jian Cheng·· 2 小时前AI 评分50
RAQ-MLA：面向多头潜在注意力缓存的角色感知量化方法
RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches
AI 导读
研究人员提出 RAQ-MLA，一种针对 Multi-Head Latent Attention（MLA）KV 缓存的角色感知量化方法，根据压缩 latent 与位置键在注意力中作用不同分别设计可逆变换。该方法无需训练即可将完整 MLA 缓存量化至约 2.5 bit/元素，理论 KV 缓存容量提升 6.5 倍，且在四个 MLA 模型的推理与编码任务上近乎无损。
正文 · 原文
Abstract:Multi-Head Latent Attention (MLA) has been adopted by recent frontier large language models. Instead of per-head keys and values, MLA caches a compressed KV latent that is up-projected into keys and values at decode time, together with a narrow positional key that carries position information into the attention score. Existing KV-cache quantization methods are designed around the keys and values of standard attention and do not account for what MLA actually caches, which costs accuracy at low bit widths. We analyze the MLA cache both in activation distribution and in how quantization error propagates to the attention output, and show that the two cached tensors play different roles: the latent mainly supplies the values, whereas the positional key dominates the attention score. We therefore propose RAQ-MLA (Role-Aware Quantization for Multi-Head Latent Attention Caches), which fits an invertible transform to each cached tensor against the error that tensor delivers to the attention output: attention-weighted value error for the latent and score perturbation for the positional key. Without training, RAQ-MLA quantizes the full MLA cache to about 2.5 bits per element, a 6.5x increase in theoretical KV-cache capacity, while remaining nearly lossless on reasoning and coding tasks across four MLA models.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.33625 [cs.AI] |
| (or arXiv:2610.33625v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33625 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chen Tianqi [view email]
[v1]
Sat, 24 Oct 2026 11:54:55 UTC (7,327 KB)
来源：arXiv cs.AI · arxiv.org