KVBuffer：面向线性注意力 IO 感知的服务方案
KVBuffer: IO-aware Serving for Linear Attention
AI 导读
KVBuffer 是一种 IO 感知的线性注意力服务机制，通过缓冲近期 key 和 value，实现解码时的 chunkwise 计算、推测解码的并行验证以及短上下文下的直接注意力计算，降低状态更新带来的内存访问开销。
正文 · 原文
Abstract:Linear attention has recently gained significant attention for long-context inference due to its constant decoding cost with respect to context length. However, existing serving systems typically serve linear attention by recurrently computing and updating a large linear attention state in every decoding step. Since the state is much larger than the per-token key and value, recurrent decoding incurs substantial memory access and becomes inefficient for serving linear attention. In this paper, we propose KVBuffer, an IO-aware serving mechanism for linear attention. By buffering recent keys and values, KVBuffer enables serving systems to compute linear attention outputs in more flexible and memory-efficient ways. For decoding, KVBuffer enables chunkwise computation, which reduces average memory access and decoding latency by deferring state updates and applying them in batch. For speculative decoding, KVBuffer verifies draft tokens in parallel and avoids storing temporary states. For short contexts, KVBuffer computes attention outputs directly from buffered keys and values, without creating or updating the linear attention state. We implement KVBuffer in SGLang for Qwen3.5 and Nemotron-3 series models. Our evaluations show that KVBuffer can speedup batch decoding by up to 1.47x and increase the maximum number of serving requests by 5x for speculative decoding when verifying four draft tokens.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.19049 [cs.LG] |
| (or arXiv:2605.19049v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19049 arXiv-issued DOI via DataCite |
Submission history
From: Longwei Zou [view email]
[v1]
Mon, 18 May 2026 19:14:52 UTC (130 KB)
[v2]
Thu, 5 Nov 2026 15:09:10 UTC (243 KB)
来源：arXiv cs.AI · arxiv.org