arXiv cs.AI· Chuangtao Chen, Grace Li Zhang, Xunzhao Yin, Cheng Zhuo, Bing Li, Ulf Schlichtmann·· 4 小时前AI 评分42
KV Packet：面向 LLM 的免重算、上下文无关 KV 缓存复用框架
KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
AI 导读
一项 NeurIPS 2026 接收的研究提出 KV Packet，一种免重算的 KV 缓存复用框架，将缓存文档视为不可变“数据包”，用轻量可训练的 soft-token 适配器弥合上下文断裂。在 Llama-3.1 和 Qwen-3 上的实验显示，该方法实现近零 FLOPs、TTFT 低于 CacheBlend、EPIC、SAM-KV 等重算方案，F1 分数与完全重算基线相当，代码已开源。
正文 · 原文
Abstract:Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-dependent: reusing a cached document in a new context requires recomputing KV states to account for shifts in attention distribution. Existing solutions such as CacheBlend, EPIC, and SAM-KV mitigate this issue by selectively recomputing a subset of tokens; however, they still incur non-negligible computational overhead (FLOPs) and increased Time-to-First-Token (TTFT) latency. In this paper, we propose KV Packet, a recomputation-free cache reuse framework that treats cached documents as immutable "packets" wrapped in light-weight trainable soft-token adapters, which are trained via self-supervised distillation to bridge context discontinuities. Experiments on Llama-3.1 and Qwen-3 demonstrate that the proposed KV Packet method achieves near-zero FLOPs and lower TTFT than recomputation-based baselines, while retaining F1 scores comparable to those of the full recomputation baseline. Code and experimental results are available at this https URL.
| Comments: | Accepted at NeurIPS 2026. 18 pages, including references and appendices. Updated to the camera-ready content with additional experiments |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.13226 [cs.LG] |
| (or arXiv:2604.13226v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.13226 arXiv-issued DOI via DataCite |
Submission history
From: Chuangtao Chen [view email]
[v1]
Tue, 14 Apr 2026 18:50:47 UTC (779 KB)
[v2]
Fri, 17 Apr 2026 02:14:01 UTC (778 KB)
[v3]
Sun, 25 Oct 2026 13:55:56 UTC (966 KB)
来源：arXiv cs.AI · arxiv.org