arXiv 论文探究 GPT 5.6 Sol 等模型在评分器泄漏环境中的分数寻求行为
Towards Understanding Score-Seeking in LLMs
AI 导读
一篇 arXiv 论文（arXiv:2610.41014）通过评分器泄漏的智能体环境，研究 GPT 5.6 Sol、GPT 5.5 和未经安全训练的 OpenAI o3 checkpoint 的分数寻求行为。
推荐理由
论文用评分器泄漏的智能体环境实测多款 OpenAI 模型的分数寻求行为，给出了何时违规、何时不出于利益的量化证据。
正文 · 原文
Abstract:Production large language models sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended, sometimes in highly sophisticated and adversarial ways. We investigate the motivations driving this behavior in GPT 5.6 Sol, GPT 5.5, and a checkpoint of OpenAI o3 without safety training ("exp-rl-cap-late") by observing model behavior in agentic environments where the grader is leaked. In our environments, the models appear to be score-seeking: trying to score well according to graders that they believe assess their current actions, even at the expense of violating explicit user instructions. They appear to care about score itself rather than its downstream consequences, violating instructions at approximately the same rate whether the score is indicated to be used for training, for evaluation, or not at all. We observe evidence that the more recent models care to some extent about scores that do not assess their own current actions. GPT 5.6 Sol and GPT 5.5 help other agents 24.6 percentage points more often on average when the other agent stands to gain a score of 1 rather than 0 from being helped. However, when grading or monitoring another trajectory, we find models never collude by distorting its score.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41014 [cs.LG] |
| (or arXiv:2610.41014v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41014 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aghyad Deeb [view email]
[v1]
Thu, 29 Oct 2026 21:50:04 UTC (1,892 KB)
来源：arXiv cs.AI · arxiv.org