arXiv cs.AI· Yezhou Cheng, Runjia Du, Zeming Liu, Qibai Chen, Hang Lyu, Yilan Wei, Yankai Zeng, Bojun Lin·· 2 小时前AI 评分43
LLM 固定预算下的高效评估框架:如何获得更窄的置信区间
Cost-Efficient LLM Evaluation: A Framework for Narrow Confidence Intervals Under Fixed Budgets
AI 导读
一项 arXiv 研究提出在固定评估预算下分配任务与重复运行的 LLM 评估框架,利用同一任务多次运行间的分歧界定估计误差,在相同置信水平下获得更窄的置信区间。理论证明当各任务重复运行结果一致时,该框架在最坏情形下达到理论最快收窄速率。在 LiveCodeBench 上,16 个模型配置中的 15 个区间宽度较均匀随机采样更窄,中位宽度减少 30.6%,每个配置预算为 1,100 个输出。
正文 · 原文
Abstract:Large language models (LLMs) play an increasingly important role in both academic research and industry. Reliable evaluation helps researchers understand model capabilities and assess progress during model development. However, evaluation can be costly, and researchers often need to assess performance within a fixed budget. Because model outputs can vary across runs, this budget must be allocated between testing different tasks and repeating selected tasks. A confidence interval quantifies the uncertainty in the resulting performance estimate. Obtaining narrower intervals at the same confidence level and evaluation cost provides a more precise assessment of model performance, helping researchers draw clearer conclusions from limited observations. We introduce a framework that allocates a fixed evaluation budget across tasks and repeated runs, with or without full task coverage. The framework then uses disagreements between runs on the same task to bound estimation error, allowing narrower confidence intervals at the same confidence level when the sampled repeats show little variability. When repeated runs on each task yield the same outcome, we prove that our framework achieves the fastest theoretically possible rates for narrowing confidence intervals in the worst case as the number of tasks and the budget for repeats grow. We establish matching lower bounds both when every task must be covered and when tasks may be skipped, even for evaluators who use earlier results to decide which run to carry out next. On LiveCodeBench, our framework yielded narrower confidence intervals on average than uniform random sampling in 15 of 16 model configurations, with a median width reduction of 30.6% at the same confidence level and the same budget of 1,100 outputs per configuration.
| Subjects: | Artificial Intelligence (cs.AI); Information Theory (cs.IT) |
| Cite as: | arXiv:2609.29140 [cs.AI] |
| (or arXiv:2609.29140v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.29140 arXiv-issued DOI via DataCite |
Submission history
From: Yezhou Cheng [view email]
[v1]
Thu, 24 Sep 2026 07:15:07 UTC (157 KB)
[v2]
Wed, 18 Nov 2026 01:01:30 UTC (55 KB)
来源:arXiv cs.AI · arxiv.org