跳到正文
arXiv cs.AI· Jiaxuan Luo, Xingguo Xu, Shanshan Wang, Yuhan Zhou, Zhen Zhang·· 2 小时前AI 评分56

CriticHack 论文揭示视觉奖励模型检验可通过而策略学错对象,并提出无需标注的修复方法

CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization

AI 导读

arXiv 论文 CriticHack 研究视觉奖励模型在机器人策略优化中的可靠性,发现成功分离与训练成功率两项常规检验可同时通过、策略却学会作用于错误对象。用 Robometer 微调扩散策略开抽屉时成功率升 10.2 个百分点,开错抽屉的失败升 10.9 个点,RoboReward-8B 与真实机器人上同样复现。

正文 · 原文

View PDF HTML (experimental)

Abstract:Learned visual reward models score a robot's execution from camera images and an instruction, and are increasingly used to fine-tune robot policies. Two checks decide whether to trust one: the reward model separates successes from failures, and task success rises during training. We show that both checks can pass while the policy learns to act on the wrong object. When a diffusion policy is fine-tuned against the reward model Robometer to open a drawer, task success rises by 10.2 percentage points. Wrong-object failures, which open a different drawer, rise by 10.9 points, whereas the same training on the simulator's task-completion signal raises success alone. A second reward model, RoboReward-8B, behaves the same way, and the effect reappears on a physical robot. This happens because of how the reward scores the policy's own failures. Most failures make little progress and score low, while a wrong-object failure looks like progress and scores well above them. Optimization shifts a policy toward outcomes that score above the executions they compete with, the executions they can take the place of. It therefore turns low-scoring failures into both successes and wrong-object failures, even when the reward model scores successes higher. This rule needs only executions of the initial policy, so it can be tested before training. On a stacking task we predicted which of three object placements Robometer would exploit, and training raised wrong-object stacks from 20% to 69% in that placement and in neither of the others. The same rule gives a check to run before optimization and a repair that needs no labels: scoring each execution again on frames re-rendered with the objects the instruction does not name put back removes the amplification on both tasks and on the robot.
Comments: 36 pages
Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2610.02527 [cs.RO]
  (or arXiv:2610.02527v2 [cs.RO] for this version)
  https://doi.org/10.48550/arXiv.2610.02527

arXiv-issued DOI via DataCite

Submission history

From: Jiaxuan Luo [view email]
[v1] Thu, 1 Oct 2026 21:57:25 UTC (1,962 KB)
[v2] Wed, 18 Nov 2026 00:11:01 UTC (1,152 KB)

来源:arXiv cs.AI · arxiv.org