arXiv cs.AI· Sichun Luo, Yi Huang, Mukai Li, Haochen Luo, Guanzhi Deng, Haibo Wang, Jian Xu, Zefa Hu, Junlan Feng, Qi Liu·· 4 小时前AI 评分36
LLM 驱动的 Agent Scaffold 进化中，评测预算应如何分配？
Where Should the Evaluation Budget Go in LLM-Driven Agent Scaffold Evolution?
AI 导读
一项 arXiv 研究考察 LLM 驱动的 agent scaffold 进化中评测预算在搜索引导与最终选择之间的分配问题。研究发现稀疏反馈可维持有效进化，而额外针对性评测能改善最终选择。所提框架在 4 个 benchmark 和 2 个进化框架上，用约一半的评测单元数即在多数设置下取得高于全量评测的平均性能。
正文 · 原文
Abstract:Large language model (LLM)-driven agent scaffold evolution improves agent workflows through iterative candidate generation and evaluation. Evaluation both guides subsequent search and determines the final scaffold, but evaluating every candidate on the full task set incurs substantial execution costs. We investigate how evaluation budgets can be allocated across these two roles. Our experiments show that sparse feedback can sustain productive evolution, while additional targeted evaluation improves final selection. Replay analyses further show that shared informative items and a common scoring reference support comparisons among closely matched candidates. Motivated by these findings, we propose \method, an evaluation allocation framework that combines sparse search-time feedback with targeted final confirmation. A short fully evaluated warm-up establishes a shared informative core and calibrates a response model whose item parameters remain fixed during search. After search, \method refits the model and applies paired successive halving, reusing accumulated observations and executing only missing evaluations. Across four benchmarks and two evolution frameworks, \method achieves higher mean performance than full evaluation in most settings while using approximately half as many evaluation cells.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Neural and Evolutionary Computing (cs.NE) |
| Cite as: | arXiv:2610.37276 [cs.LG] |
| (or arXiv:2610.37276v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37276 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sichun Luo [view email]
[v1]
Tue, 27 Oct 2026 14:06:26 UTC (622 KB)
来源：arXiv cs.AI · arxiv.org