arXiv cs.AI· Rose N. Kwon, Samy Mammeri, Sabyasachi Sahoo, Christian Gagn\'e, Thomas Jiralerspong·· 2 小时前AI 评分45
隐状态相似度可预测 inoculation prompting 后的再激发
Hidden-State Similarity Predicts Re-Elicitation After Inoculation Prompting
AI 导读
一篇 arXiv 论文研究 inoculation prompting 抑制有害行为后的再激发机制。研究发现,与 inoculation 提示词的文本相似度并不足以预测触发效果,当评估提示词诱导出与 inoculation 上下文相似的激活状态时,被抑制行为更可能重现。这表明激活空间分析可帮助判断被抑制行为在评估时是否仍可被激发。
正文 · 原文
Abstract:Fine-tuning on narrow harmful tasks can cause emergent misalignment, where models generalize harmful behavior beyond the training distribution. Inoculation prompting can reduce this effect by explicitly eliciting the undesired behavior during training, but recent work shows that the behavior can reappear when evaluation prompts contain cues from the training context. We study what makes such prompts effective triggers. We find that, surprisingly, textual similarity to the inoculation prompt is an incomplete predictor. Instead, prompts are more likely to re-elicit suppressed behavior when they induce activation states similar to those produced by the inoculation context. These findings advance our understanding of how inoculation prompting modulates conditional misalignment, and suggest that activation space analysis can help identify when suppressed behaviors remain accessible under eval-time prompts.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23096 [cs.AI] |
| (or arXiv:2611.23096v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23096 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rose Nayoung Kwon [view email]
[v1]
Tue, 17 Nov 2026 17:20:33 UTC (402 KB)
来源:arXiv cs.AI · arxiv.org