arXiv cs.AI· Cheng-Yu Hsieh, Ting-Yao Hu, Sanjoy Chowdhury, Oncel Tuzel, Raviteja Vemulapalli·· 3 小时前AI 评分43
JudgeMix：面向 LLM Agent 的评审引导混合 rollout 数据合成
JudgeMix: Judge-Guided Mixed-Rollout Data Synthesis for LLM Agents
AI 导读
一项研究提出 JudgeMix，一种由评审（judge）引导的数据合成框架，用于训练 LLM agent。该框架基于逐步语义评估动态交错目标模型与专家干预：仅在目标模型遇到困难时调用专家指导，并用评估结果对各步骤的专家监督重新加权，从而在保持目标模型状态分布的同时修正错误。
正文 · 原文
Abstract:Supervised fine-tuning (SFT) on expert trajectories is a standard paradigm for training LLM agents. However, it suffers from a well-known training-inference distribution mismatch, calling for target-aware data synthesis strategies that ground the training data in the target model's native state distribution. To bridge this gap, emerging approaches incorporate the expert either to guide mixed rollouts alongside the target model, or to provide dense step-wise supervision. Nonetheless, existing methods apply expert assistance blind to the target model's inherent behavior: they either disrupt the rollout using sub-optimal random intervention heuristics, or they enforce indiscriminate uniform supervision that unnecessarily overwrites the model's already-correct actions. We introduce JudgeMix, a judge-guided data synthesis framework that dynamically interleaves the target model and expert based on step-wise semantic evaluations. JudgeMix queries expert guidance precisely when the target model struggles, ensuring high trajectory success rates while keeping the state distribution aligned with the target model. It additionally uses the judge evaluation to reweight expert supervision on different steps, preserving model's inherent valid behavior while correcting its errors. On two agentic tool-use benchmarks, AppWorld and AutomationBench, JudgeMix demonstrates consistent superior performance to existing methods, with improvements up to 10 pass@1 points.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.16174 [cs.LG] |
| (or arXiv:2611.16174v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16174 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Cheng-Yu Hsieh [view email]
[v1]
Thu, 12 Nov 2026 02:03:23 UTC (15,542 KB)
来源：arXiv cs.AI · arxiv.org