arXiv cs.AI· Mufei Li, Nikki Lijing Kuang, Zhuoer Wang, Samson Koelle, Jie Yuan, Manqing Mao, Hong Wang, Yanjun Lin, James Feng, Daniel Edmiston, Pan Li, Wei Niu·· 3 小时前AI 评分31
多轮 On-Policy 蒸馏中如何选取成功轨迹作参考：基于动作序列对齐的方法
Which Successes Make Good References? Action-Sequence Alignment for Multi-Turn On-Policy Distillation
AI 导读
一篇 arXiv 论文提出将多轮工具调用智能体 on-policy 蒸馏中的成功参考轨迹选择建模为自由目标最优传输问题，用最长公共子序列（LCS）作为动作序列兼容性的可观测代理。在 ALFWorld 和 τ³-bench 航空域实验中，LCS 是唯一在 ALFWorld 总分和全部三项航空指标上都优于无参考 OPD 的参考选择器，且航空域训练的模型无需进一步优化即可迁移到零售域并提升全部三项指标。
正文 · 原文
Authors:Mufei Li, Nikki Lijing Kuang, Zhuoer Wang, Samson Koelle, Jie Yuan, Manqing Mao, Hong Wang, Yanjun Lin, James Feng, Daniel Edmiston, Pan Li, Wei Niu
Abstract:On-policy distillation (OPD) reduces train-inference distribution mismatch by supervising a student on its own trajectories. For multi-turn tool-use agents, conditioning the teacher on a validated same-task successful trajectory as a reference can provide useful hindsight, but different successes may reach incompatible interaction states and offer misleading supervision. We cast successful-reference selection as a free-target optimal transport problem, which reduces to assigning each failed trajectory its most compatible successful reference. Because latent state paths are hidden, we use action-sequence alignment as an observable proxy for compatibility and instantiate it with longest common subsequence (LCS), which identifies a shared ordered backbone of actions while allowing recoverable detours. This perspective makes trajectory compatibility the central criterion for reference selection in multi-turn OPD. Experiments on ALFWorld and the airline domain of $\tau^3$-bench show that LCS is the only evaluated reference selector to improve over reference-free OPD on both ALFWorld overall scores and all three airline metrics. Without further optimization, the airline-trained models also transfer to the retail domain, where LCS improves all three metrics.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.13135 [cs.LG] |
| (or arXiv:2611.13135v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13135 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mufei Li [view email]
[v1]
Mon, 9 Nov 2026 19:18:25 UTC (4,005 KB)
来源：arXiv cs.AI · arxiv.org