Goodhart in the Planner：基于采样的规划搜索如何利用人类偏好学到的奖励
Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences
AI 导读
一项研究测量了机器人基于人类偏好奖励模型做 MPPI 搜索对真实目标的影响：在 HalfCheetah、Hopper 及 Meta-World 抽屉开启、盘子推滑任务上，更难的搜索使真实回报最多下降 76%，而 held-out 标签准确率仍保持 0.75-0.97。
正文 · 原文
Abstract:Robots increasingly plan by searching against a reward model learned from human preferences; we measure what this search does to the true objective. On HalfCheetah and Hopper with crowd-sourced labels, and on Meta-World drawer-open and plate-slide with synthetic labels and blind labels from the two authors, harder MPPI search cuts true return by up to 76% while held-out label accuracy stays at 0.75-0.97: with human labels the drawer opens in 0 of 20 runs, against 10 of 10 for the cloned policy the planner starts from. The effect persists with a learned dynamics model and with an image scorer. True return roughly follows how far the planner's weights move from its proposals ($\sqrt{\mathrm{KL}}$), and best-of-$N$ action selection with a verifier trained on synthetic labels falls below a single noisy action. We prove that $\chi^2$-regularized MPPI at a temperature set by the scorer's error spread on the proposals is worst-case optimal, that no gain can be certified once that spread exceeds the scorer's own, and that an episode's return splits exactly into per-step terms, only some of which the guarantee covers. Calibrated from 100 proposal pairs ranked by true return (an oracle), this rule is the safest fix we test: it keeps on average 86-94% of the cloned policy's return on drawer-open and plate-slide, though it is cautious on window-open, where search helps. Calibrated from a person's own comparisons, it cannot detect a bias the scorer shares with that person; a second person, or correcting for pairs people cannot tell apart, limits the damage.
| Comments: | 21 pages, 4 figures. Equal contribution. Code and results will be released |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.09670 [cs.RO] |
| (or arXiv:2611.09670v2 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2611.09670 arXiv-issued DOI via DataCite |
Submission history
From: Soovadeep Bakshi [view email]
[v1]
Fri, 6 Nov 2026 16:22:17 UTC (514 KB)
[v2]
Thu, 12 Nov 2026 07:40:41 UTC (517 KB)
来源：arXiv cs.AI · arxiv.org