AI 对齐研究新论文：给出“灾难性性能”的形式化定义
A Definition of Catastrophic Performance
AI 导读
arXiv 论文提出“灾难性性能”的正式定义，以区分 AI 目标错设导致的次优结果与灾难性后果。该定义基于两个参考值：baseline value 表示不部署智能体时的价值，orthogonal value 表示与人类偏好无关轨迹的价值，后者通常远差于前者。作者以人类活动改善环境状态的例子说明，灾难性性能即 AI 智能体抵消人类活动所带来的收益。
正文 · 原文
Abstract:A central challenge in AI is specifying objectives that align with human preferences. While existing theoretical results elucidate how misspecification impacts performance, they typically do not distinguish between merely suboptimal and catastrophic outcomes. To enable rigorous analysis of catastrophic risk from AI misalignment, we introduce a formal definition of catastrophic performance. Our definition is based on two reference values. The first, baseline value, captures the value attained if no agent is deployed. The second, orthogonal value, captures the value of a trajectory that is unrelated to human preferences, in a sense we describe. Orthogonal value is typically far worse than baseline value. We interpret these reference values through examples in which past human activity has altered the state of the environment to human benefit. Starting at the resulting state, an AI agent is deployed with the goal of accelerating further progress. Catastrophic performance represents this agent's undoing of benefits afforded by human activity.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.06219 [cs.AI] |
| (or arXiv:2611.06219v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06219 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alex Infanger [view email]
[v1]
Wed, 4 Nov 2026 09:30:13 UTC (1,043 KB)
来源：arXiv cs.AI · arxiv.org