arXiv cs.AI· Zhiliang Chen, Alfred Wei Lun Leong, Shao Yong Ong, Apivich Hemachandra, Gregory Kang Ruey Lau, Chuan-Sheng Foo, Zhengyuan Liu, Nancy F. Chen, Bryan Kian Hsiang Low·· 5 小时前AI 评分35
JoBS：重新思考用贝叶斯优化协同调优 LLM 训练配置
Rethinking Bayesian Optimization for Co-Optimizing LLM Training Configurations
AI 导读
arXiv 论文提出 JoBS，一种基于贝叶斯优化（BO）的方法，可协同优化 LLM 微调的数据与模型配置，而无需每次迭代都完整训练模型。JoBS 先用部分预算学习受 scaling law 启发的性能预测器，从少量训练步数估计完全训练后的性能，再用剩余预算运行 BO，从而在相同预算下探索更多配置。
正文 · 原文
Abstract:Fine-tuning an LLM to maximize performance on a downstream task requires finding the optimal data and model configuration. This is a black-box optimization problem that Bayesian optimization (BO) addresses by sequentially evaluating training configurations and using observed performance feedback to adaptively guide the search towards better configurations. However, directly applying BO to the joint data-model space suffers from two fundamental challenges: the prohibitive cost of full LLM training at each iteration, and the large number of BO iterations required for effective exploration due to the high problem dimensionality. This paper introduces JoBS, a BO-based approach that co-optimizes data and model configurations without requiring a full training run at every iteration. JoBS allocates an initial portion of the optimization budget to learn a scaling-law-inspired performance predictor that estimates fully trained LLM performance from only a small number of training steps. The remaining budget is then used to run BO with this predictor, eliminating the need for full training runs and enabling JoBS to explore significantly more data and model configurations within the same budget. We analyze JoBS's average regret and derive the optimal budget allocation between predictor learning and BO iterations. Empirically, JoBS outperforms independent data and model optimization methods and existing multi-fidelity BO baselines across a diverse set of downstream tasks. Our code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.08351 [cs.LG] |
| (or arXiv:2602.08351v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.08351 arXiv-issued DOI via DataCite |
Submission history
From: Zhiliang Chen [view email]
[v1]
Mon, 9 Feb 2026 07:33:40 UTC (7,986 KB)
[v2]
Thu, 19 Feb 2026 01:32:53 UTC (7,986 KB)
[v3]
Wed, 28 Oct 2026 07:55:31 UTC (7,655 KB)
来源：arXiv cs.AI · arxiv.org