arXiv cs.AI· Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese, Wesley Harris·· 2 小时前AI 评分37
WAKMan：负载感知的 KV cache 管理器
WAKMan: Workload Aware KVCache Manager
AI 导读
WAKMan 是一种负载感知的 KV cache 管理器，可在不识别请求类型、不读取提示词的情况下，根据实时流量选择 LLM 前缀缓存的淘汰策略。在 20 种负载混合测试中，单一策略各有优劣：vLLM 默认的 LRU 在 agentic 重度负载上比 S3FIFO 高 12% 缓存命中，但其他场景最多落后 22%。
正文 · 原文
Abstract:Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost. With limited memory resource, the KV cache eviction policy becomes a decisive factor in determining the overall benefits of prefix caching. Production traffic may have single-turn, multi-turn, agentic, or a mixture of any combination of these request types. Our findings indicate that agentic steps return within seconds and favor recency, while popular prompts and long conversations return after many other requests and favor frequency. Across 20 mixes, no single policy is always best: vLLM's default strategy LRU provides 12% higher cache hits as compared to the frequency-oriented S3FIFO on agentic-heavy mixes but trails it by up to 22% elsewhere, and even a single workload parameter can flip the winner even for any particular mix. We present WAKMan, a workload-aware KV cache manager that picks its eviction policy from live traffic without knowing request types or reading prompts. It improves hit rate over LRU by 5 percentage points on average, with the lowest mean and worst-case optimality gap of all policies.
| Comments: | Work done during internship at Capital One |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.36355 [cs.AI] |
| (or arXiv:2610.36355v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36355 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vaibhavi Lokegaonkar [view email]
[v1]
Mon, 26 Oct 2026 22:19:37 UTC (502 KB)
来源：arXiv cs.AI · arxiv.org