arXiv cs.AI· Ruth Wan Theng Chew, Zhiliang Chen, Apivich Hemachandra, Bryan Kian Hsiang Low·· 4 小时前AI 评分41
BoLT：面向昂贵 LLM 任务的黑盒优化研究开放基准
BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks
AI 导读
研究者发布 BoLT，首个面向 BBO 社区的 LLM 专用基准，旨在让无法承担真实 LLM 训练与推理成本的研究者也能开展黑盒优化研究。基准覆盖多保真、多目标、异方差噪声与高维搜索空间等真实 LLM 优化问题，每个问题基于数千次真实 LLM 实验数据，通过轻量级代理模型实现完全可复现。
正文 · 原文
Abstract:Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is often approached heuristically in practice, leading to potentially suboptimal outcomes. By framing them as noisy, expensive, and derivative-free optimization problems, Bayesian optimization (BO) and other black-box optimization (BBO) methods offer a promising yet underexplored direction for principled, sample-efficient methods. However, LLM training and inference costs are prohibitively high for most of the BBO research community, and new methods are often only evaluated on synthetic test functions and small-scale datasets that fail to capture the challenges of modern LLM optimization problems. This impedes the development of BBO methods and makes it difficult to assess their effectiveness on modern LLM tasks. We introduce BoLT, the first LLM-centric benchmark that democratizes LLM research for the BBO community. BoLT is released at this https URL. BoLT covers broad and well-motivated LLM optimization problems, involving multi-fidelity, multi-objective, heteroscedastic noise, and high-dimensional search spaces. Each problem in BoLT is grounded in real experimental data and made fully reproducible and accessible through lightweight surrogate models fitted to the results of thousands of real LLM experiments. We benchmark BoLT against an extensive range of BO and BBO methods, showing that selected BO methods consistently outperform others across tasks and highlighting gaps in existing BBO methods on LLM tasks, underscoring the need to modernize benchmarks for the BBO community.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.17000 [cs.LG] |
| (or arXiv:2605.17000v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.17000 arXiv-issued DOI via DataCite |
Submission history
From: Zhiliang Chen [view email]
[v1]
Sat, 16 May 2026 13:53:44 UTC (2,089 KB)
[v2]
Tue, 27 Oct 2026 03:52:46 UTC (3,510 KB)
来源：arXiv cs.AI · arxiv.org