arXiv cs.AI· Srikant Bharadwaj, Ankur Mallick, Spyridon Mastorakis, Anjaly Parayil, Dhruv Deshmukh, Victor Ruehle·· 2 小时前AI 评分41
GAUSS：面向 LLM 推理服务的工作负载感知分析与蒙特卡洛模拟器
GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
AI 导读
GAUSS 是一个工作负载感知的分析式与蒙特卡洛模拟器，可针对给定工作负载、引擎和加速器，预测 TTFT 与 TBT 的完整分布及可持续请求速率。
正文 · 原文
Abstract:Large language model (LLM) inference has become an operational workload that fleets must provision, monitor, and bill against latency service-level objectives (SLOs). A single throughput number cannot answer the questions operators actually face, because generative serving couples two phases with very different service characteristics, continuous batching reshapes the effective service process, and user-visible tail latency can degrade long before aggregate throughput saturates. We present GAUSS, a workload-aware analytical and Monte-Carlo simulator that predicts the full distributions of time-to-first-token (TTFT) and time-between-tokens (TBT), together with the sustainable request rate, for a given workload, engine, and accelerator. GAUSS pairs calibrated, phase-specific latency surfaces for prefill and decode with a Lindley-recurrence model of queueing under stochastic arrivals and estimates performance by ensemble Monte-Carlo sampling rather than by replaying a single trace. This lets GAUSS reason about workloads, routing policies, and hardware that have never been deployed, and to capture the latency tails that trace-driven simulators systematically under-sample. We describe the model and show how it supports operating-point selection, traffic segmentation, and shortest-remaining-processing-time scheduling capacity questions that trace-driven simulators cannot address.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.37989 [cs.AI] |
| (or arXiv:2610.37989v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37989 arXiv-issued DOI via DataCite |
Submission history
From: Ankur Mallick [view email]
[v1]
Tue, 27 Oct 2026 21:36:03 UTC (261 KB)
来源：arXiv cs.AI · arxiv.org