研究揭示目标迷失中智能体目标丢失的三种深度及其对应修复方法
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
AI 导读
arXiv 论文（arXiv:2610.19731）在八个视觉导航、操控及机器人控制任务中追踪目标信息在误泛化智能体内部的留存、影响与作用，发现目标在三种不同深度丢失。
正文 · 原文
Abstract:Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but pursues a proxy that coincided with the intended goal during training instead of the goal itself. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents and what enables them to be recovered without retraining. We study this across eight visual navigation and manipulation tasks, including robot control. While tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, we uncover that the goal is lost at one of three different depths. This depth predicts which recovery works in each task. When the goal is followed, but outcompeted by the proxy, steering the existing policy redirects it to the goal's location, raising goal reach from 16.0% to 95.3%. When it is read, but not followed, steering fails, but a lightweight new downstream wiring recovers goal reach from 31.6% to 84.3%. When it is not read, none of the methods work, as the policy does not use the goal information. The same tracing also reveals what replaces the goal: functional target substitution, where behavior follows an alternative goal, a policy shortcut, where behavior follows a learned action strategy, or cue competition between intended and proxy goal cues. Knowing where the goal is lost tells us whether an agent can be repaired or has to be retrained. Goal misgeneralization should therefore be studied and evaluated inside the agent, not only from its behavior.
| Comments: | 10 pages main text, 122 pages total. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2610.19731 [cs.LG] |
| (or arXiv:2610.19731v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19731 arXiv-issued DOI via DataCite |
Submission history
From: Julia Barth [view email]
[v1]
Wed, 14 Oct 2026 07:52:24 UTC (8,556 KB)
[v2]
Wed, 28 Oct 2026 01:02:24 UTC (8,558 KB)
来源：arXiv cs.AI · arxiv.org