跳到正文
arXiv cs.AI· Ishaan Datta, Alind Khare, Koushik Kumar Dey, Ratnesh Jamidar, Anand Kumar Singh·· 4 小时前AI 评分36

VelServe:用尾延迟校准调度解决 LLM 推理中的解码抖动

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

AI 导读

arXiv 论文提出 VelServe 调度器,利用解码步骤迭代延迟的实测尾部数据校准每步 prefill 负载,为误预测和主机停顿预留余量,以降低 LLM 推理中的解码抖动并满足 TTFT 与 TBT 的 SLO。

正文 · 原文

View PDF HTML (experimental)

Abstract:Interactive LLM services must meet latency service-level objectives (SLOs) for the time to first token (TTFT) and the time between tokens (TBT), as streaming and soft real-time applications require predictable token delivery. We identify decode jitters in both aggregated and disaggregated serving. A jitter delays every request in the batch, and long responses span hundreds of steps, so even rare overruns cause many SLO violations. In aggregated serving, SLO-aware schedulers set each step's prefill budget until its predicted latency just meets the TBT target, leaving no room for mispredictions or host stalls. Disaggregation meets average TBT targets at a cost in capacity, yet the steps that admit new requests still run late.
We present \sysname, a scheduler that calibrates how much prefill work each decode step carries using the measured tail of its own iteration latency. The SLO determines how rare a late step must be, and \sysname leaves enough headroom in each iteration to prevent overruns and meet the target decode latency. This headroom slows prefill, so \sysname serves the shortest waiting prompt first. When TTFT deadlines grow with prompt length, the long prompts it delays have the most time to spare. Under disaggregation, it spaces admissions so that no request waits on more slow admission steps than its P99 target allows. On three attention families and varying workloads, \sysname sustains 2.4--2.9$\times$ the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4$\times$ on DeepSeek-V2-Lite. With prefill and decode disaggregated on eight GPUs, \sysname meets the SLO for more than twice as many requests as state-of-the-art baselines.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.18392 [cs.AI]
  (or arXiv:2611.18392v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.18392

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ishaan Datta [view email]
[v1] Fri, 13 Nov 2026 16:34:37 UTC (2,497 KB)

来源:arXiv cs.AI · arxiv.org