arXiv cs.AI· Xinrui Zhong, Zhiming Wang, Yongqi Liu, Zhaohui Wang, Shanglin Yang, Zikun Li, Zhuoming Chen, Beidi Chen·· 3 小时前AI 评分46
ServerBench：让编码智能体设计专用 LLM 服务系统的基准
ServerBench: How can your coding agents design specialized LLM serving systems?
AI 导读
ServerBench 是一个评测编码智能体设计完整专用 LLM 服务系统能力的基准，目标指标为"每美元延迟合格 token 价值"，并通过状态化回放保留会话依赖、独立验证检查提交系统。
正文 · 原文
Abstract:Large language model (LLM) coding agents develop software through experimentation, offering an opportunity to design the serving systems supporting their own operation. Effective serving requires coordinating implementation and operating decisions under session dependencies, latency constraints, and correctness requirements. We introduce ServerBench, a benchmark for agentic design of complete, specialized LLM serving systems. Agents experiment against a common objective, dollars per dollar: latency-qualified token value per dollar of GPU cost. Stateful replay preserves session dependencies, and independent verification checks the submitted system. Across 11 coding-agent models and three deployments, Claude Sonnet 5.5 achieves 2.97 dollars per dollar on Gemma E4B, $4.87\times$ the vLLM reference; 22 of 33 runs yield verified systems, and 13 exceed the strongest baseline. Our analysis identifies three findings: high-value designs coordinate capacity improvements with load selection; agents interpret performance feedback through workload-relevant experiments; and reliable delivery requires verification of the execution paths introduced by optimization. Overall, ServerBench supports systematic study of how agents translate service objectives and feedback into effective, verified serving systems for given task specifications.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06841 [cs.AI] |
| (or arXiv:2611.06841v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06841 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhuoming Chen [view email]
[v1]
Wed, 4 Nov 2026 17:59:07 UTC (736 KB)
来源：arXiv cs.AI · arxiv.org