跳到正文
arXiv cs.AI· Zhuohan Cai, Zhangxuan Gu, Runze Li, Beitong Zhou, Ming Wen, Hongbo Yu, Shuheng Shen, Changhua Meng·· 56 分钟前AI 评分43

PACT:通过保护正确计划与修复动作训练提升 GUI 智能体执行一致性

PACT: Plan-Action Consistency Training by Preserving Correct Plans and Repairing Actions in GUI Agents

AI 导读

PACT(Plan-Action Consistency Training)是一种针对 GUI 智能体的训练框架,通过保护已验证计划并对齐修复错误动作,解决结果监督强化学习中的信用分配冲突。

正文 · 原文

View PDF HTML (experimental)

Abstract:GUI agents can formulate a correct immediate plan in their reasoning yet emit an action that fails to execute it, creating an execution gap. This recurring mismatch undermines reliable task completion, as a single execution error can derail a multi-step workflow. In outcome-supervised reinforcement learning, broadcasting a negative response-level advantage across all generated tokens risks penalizing valid planning together with erroneous execution. To address this credit-assignment conflict, we introduce PACT (Plan-Action Consistency Training), a training framework that combines reference-aligned plan protection and plan-aligned action repair without altering the original scalar action reward. Using annotated reference actions to identify execution gaps, PACT protects the verified plan by masking negative policy-gradient contributions across its token span. Keeping the generated reasoning unchanged as context, it then favors the reference action over the emitted erroneous action through a preference surrogate with stop-gradient pair weights. Compared with the outcome-only baseline, PACT raises AndroidWorld task success from 60.34% to 68.97% (+8.63 pp), while improving rule-based plan-action consistency on AndroidControl High from 83.71% to 89.74% (+6.03 pp). Adapting PACT to desktop agents also improves task success from 40.92% to 44.17% (+3.25 pp) on the full 369-task OSWorld benchmark. Together, these results establish PACT as an effective credit-assignment framework for bridging execution gaps and improving task completion across mobile and desktop GUI environments.
Comments: 21 pages, 11 figures, including supplementary material
Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
Cite as: arXiv:2611.22591 [cs.AI]
  (or arXiv:2611.22591v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22591

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zhuohan Cai [view email]
[v1] Tue, 17 Nov 2026 11:14:59 UTC (6,188 KB)

来源:arXiv cs.AI · arxiv.org