arXiv cs.AI· Yifan Wang, Zhenyi Gai, Yiwen Zhao, Mingxuan Jiang, Keyang Chen, Zeping Li, Hongfeng Chai, Yixin Cao·· 2 小时前AI 评分29
arXiv 论文提出诊断策略更新下技能效用变化的方法
Before the Next Rollout: Diagnosing Skill Utility Shifts from Policy Updates
AI 导读
一项 arXiv 研究提出，策略更新会改变智能体技能的边际效用，使基于旧执行轨迹的技能归因失效。研究者在更新前的固定上下文中测量技能与对照条件的 token 偏好对比变化，结合优势估计得到效用变化的诊断读出，仅用前向传播即可计算，无需更新后的额外轨迹评估。实验显示读出能有效识别效用下降的技能，将其用于技能进化可提升下游性能并降低策略与编辑器 token 消耗。
正文 · 原文
Abstract:Skill-evolution methods typically analyze execution trajectories to guide agent skill refinement. However, policy updates can change the behavioral effects and marginal utility of an unchanged skill. Attribution based on earlier trajectories may therefore no longer reflect the skill's contribution under the updated policy. Moreover, trajectory failures can arise from multiple factors, making individual skill contributions difficult to isolate. We evaluate whether policy-update signals contain predictive information about changes in skill marginal utility. At fixed contexts from pre-update training rollouts, we measure how policy updates change the contrast in token preferences between skill and control conditions. Combining these interaction shifts with advantage estimates yields reward-directed readouts for diagnosing utility changes. The readouts reuse existing training rollouts and require only forward passes for single-step action scoring, without additional post-update rollout evaluation. Our experiments show that policy updates can alter skill marginal utility and that policy-update readouts carry diagnostic information for identifying skills with declining utility. Integrating readout-guided prioritization into skill evolution improves downstream performance while reducing policy and editor token consumption.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39358 [cs.AI] |
| (or arXiv:2610.39358v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39358 arXiv-issued DOI via DataCite |
Submission history
From: Yifan Wang [view email]
[v1]
Thu, 29 Oct 2026 04:50:24 UTC (1,628 KB)
来源：arXiv cs.AI · arxiv.org