arXiv cs.AI· Yingjian Chen, Lang Zhou, Zixin Zhao, Pingfan Li, Weishi Zheng, Zhilin Zhao·· 4 小时前AI 评分40
ConCal：基于续段共享的动作价值校准方法，改进长程智能体强化学习
Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning
AI 导读
一篇 arXiv 论文提出 ConCal（Continuation Calibration），一种递归续段价值估计方法，利用 rollout batch 中已有的续段证据校准动作价值，缓解长程智能体强化学习中的局部信用分配失真，且无需额外 rollout。
正文 · 原文
Abstract:Reinforcement learning for long-horizon agents relies on feedback from sequences of interactions, making credit assignment to individual actions difficult. Existing methods such as GiGPO compare actions at shared states to compute relative advantages, but estimate each action's value from a single sampled continuation. The resulting local credit remains sensitive to decisions made later in the trajectory. To address this problem, we propose ConCal (Continuation Calibration), a recursive continuation-value estimator that incorporates continuation evidence already present in a rollout batch. ConCal matches visits from the same task by their incoming state-action-state transitions, combines shared continuation evidence with each visit's own transition estimate, and propagates calibrated values backward to refine earlier action-value estimates. This reduces dependence on individual sampled continuations and mitigates local credit distortions without additional rollouts. Held-out return diagnostics show lower error in within-group value differences and higher agreement with reference action orderings than estimates based on individual continuations. With Qwen2.5-1.5B-Instruct, ConCal improves success rates over GiGPO by 3.39 and 5.86 percentage points on ALFWorld and WebShop, respectively. Our code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.37195 [cs.LG] |
| (or arXiv:2610.37195v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37195 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yingjian Chen [view email]
[v1]
Tue, 27 Oct 2026 13:16:08 UTC (2,991 KB)
来源：arXiv cs.AI · arxiv.org