通过社会化实现对齐:选择压力下的权衡学习(arXiv 论文)
Alignment by Socialization: Learning Tradeoffs Under Selection
AI 导读
一篇 arXiv 论文通过进化选择模型研究 AI 智能体如何在社会规范与任务完成之间做权衡。论文比较三种策略:只重任务的策略在仅奖励完成度的环境中繁荣,但在行为规范重要时失败;固定线性策略随任务利害上升偏向完成任务;学习型策略则收敛到环境奖励的权衡,当奖励由人类提供时称为“社会化”。
正文 · 原文
Abstract:Artificial intelligence (AI) agents are expected to complete tasks while following rules of conduct, but these objectives can conflict. We study how evolutionary selection shapes an agent's choices between them, and whether experience can make those choices reflect human judgments and withstand subsequent pressures. In our model, copies of an agent face a common uncertain reward state, and only those taking the rewarded action reproduce. We show that long-run growth is maximized when each action is taken as often as the environment rewards it. We compare three policies. A task-only policy thrives in a sandbox rewarding completion alone but fails when conduct matters. A fixed linear policy increasingly favors task completion as its stakes rise, even where humans never reward it. A learned policy instead converges to the tradeoffs rewarded by its environment; when humans supply the rewards, we call this socialization. Accumulated experience makes socialization harder to reverse under subsequent task-only training, although violations can arise before the prevailing disposition changes, and finite memory weakens this protection. When a single violation can end a deployment, a hard prohibition can prevent it, but a mistaken prohibition can also be costly. Experiments with small neural networks illustrate how training targets and memory shape these outcomes. Our results suggest that alignment depends not only on the objectives assigned to agents, but also on the environments that shape and sustain their behavior.
| Subjects: | Artificial Intelligence (cs.AI); Populations and Evolution (q-bio.PE); General Finance (q-fin.GN) |
| Cite as: | arXiv:2611.18531 [cs.AI] |
| (or arXiv:2611.18531v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.18531 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Andrew Lo [view email]
[v1]
Fri, 13 Nov 2026 18:43:24 UTC (261 KB)
来源:arXiv cs.AI · arxiv.org