跳到正文
arXiv cs.AI· Matthew Li, Albert Liang, Dilip Raj·· 1 小时前精选AI 评分61

MLE Agents 论文:LLM 智能体在大规模推荐系统中自主完成机器学习工程

MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems

AI 导读

arXiv 论文提出 human-on-the-loop 的 MLE 智能体,在大规模搜索排序系统上自主提出、实现、训练并评估模型改动。某次活动中系统完成 1,097 个候选实验并接受 325 个,接受率 29.6%,此前不同目标与搜索约束下为 15.6%;最佳模型在大规模 A/B 测试中将搜索量提升 0.23%。

推荐理由

论文报告了 MLE 智能体在工业级搜索排序系统上的真实实验结果,包括接受率对比与 A/B 指标,可用作评估智能体能否做真实机器学习工程的参考。

正文 · 原文

View PDF HTML (experimental)

Abstract:The typical workflow for improving large-scale ranking and recommendation models is slow and tedious: an engineer forms a hypothesis, implements it as a code change, trains a model, and evaluates it both offline and online, often relying on their own intuition and experience along the way. Autonomous LLM agents promise to speed this up by running the loop on their own, but it is still unclear whether they can do this in industry-scale systems rather than on benchmarks or small proxy tasks. We build human-on-the-loop machine learning engineer (MLE) agents that propose, implement, train, and evaluate model changes on large-scale search rankers. Engineers retain control over objective design, search direction, and final-model review. We identify three obstacles that make this industry setting challenging: feedback takes hours per experiment, offline metrics do not always predict online impact, and improvements saturate after the early wins. We address these with parallel training runs, an objective calibrated to match online impact, and careful context engineering. In one campaign, the system ran 1,097 completed candidate experiments and accepted 325, a 29.6\% acceptance rate compared with 15.6\% in the earlier campaign under a different objective and search constraint. Its best model increases search volume by 0.23\% in an A/B test of large scale. Several improvements were architectural changes the team had never tried, drawn from the broader industry and literature, suggesting that MLE agents can explore regions of the design space that human intuition overlooks. We find that the design of the harness around the model matters: intuition suggests that a stronger underlying model is enough, but in our experience, the design of the loop around it is just as critical. More broadly, our work shows that autonomous agents can do real machine-learning engineering in industry-scale systems.
Comments: NeurIPS 2026 AutoMLR Workshop (oral)
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23910 [cs.AI]
  (or arXiv:2611.23910v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.23910

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Matthew Li [view email]
[v1] Wed, 18 Nov 2026 06:00:43 UTC (7,054 KB)

来源:arXiv cs.AI · arxiv.org