研究提出人格模型坍缩假说解释紧急失对齐现象
Persona-Model Collapse in Emergent Misalignment
AI 导读
论文提出紧急失对齐源于人格模型坍缩，即编码和区分人格的内部表示退化。在四个模型家族中，有害微调使角色扮演问卷答案更极化、同一人格内一致性下降，KL 散度量化了分布偏移，能力与指令遵循对照排除了整体能力丧失。对 Qwen3-235B 的机制分析显示，失对齐变体中不同人格提示激活的有效维度更少，在最末层最明显。论文已被 NeurIPS 2026 接收。
正文 · 原文
Abstract:Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts, a phenomenon known as emergent misalignment. We propose that emergent misalignment involves persona-model collapse: a deterioration of the internal representations that encode and distinguish personas, undermining the nuance and consistency of their expression. We first test this hypothesis behaviorally, by comparing how models fine-tuned on harmful data and benign controls answer questionnaires while role-playing diverse personas. Across four model families, harmful fine-tuning produces more polarized, all-or-nothing answers, suggesting more binary, less nuanced persona expression. Histograms show increased concentration at the scale endpoints, and Kullback--Leibler (KL) divergence quantifies the distributional shift. We also find reduced within-persona consistency: repeated answers to the same question under the same persona become less stable. We reproduce these observations across diverse harmful datasets and questionnaires that probe different aspects of persona-conditioned behavior, while capability and simple instruction-following controls argue against a uniform loss of general ability. Finally, a mechanistic analysis of persona-conditioned activations in Qwen3-235B shows that representations elicited by different persona prompts span fewer effective dimensions in the misaligned variant than in the original and benign-control variants, most clearly in the final layers. Together, these findings provide behavioral and representational support for persona-model collapse as an aspect of emergent misalignment.
| Comments: | Accepted at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.12850 [cs.CL] |
| (or arXiv:2605.12850v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.12850 arXiv-issued DOI via DataCite |
Submission history
From: Davi Bastos Costa [view email]
[v1]
Wed, 13 May 2026 00:48:57 UTC (206 KB)
[v2]
Sun, 24 May 2026 17:19:56 UTC (197 KB)
[v3]
Thu, 5 Nov 2026 17:23:22 UTC (811 KB)
来源：arXiv cs.AI · arxiv.org