arXiv cs.AI· Seanie Lee, Sangwoo Park, Yumin Choi, Gyeongman Kim, Minki Kang, Jihun Yun, Sung Ju Hwang, Dongmin Park, Jongho Park·· 4 小时前AI 评分39
ThinkSafe：面向推理模型的自生成安全对齐方法
ThinkSafe: Self-Generated Safety Alignment for Reasoning Models
AI 导读
ThinkSafe 是一种无需外部教师模型的安全对齐方法，将安全再对齐形式化为 KL 投影，从冻结的学生模型自身生成安全对齐数据：通过拒绝引导引出安全推理，再用安全守卫过滤后进行离线微调。在 DeepSeek-R1-Distill 和 Qwen3 上的实验显示安全性显著提升且推理性能保持竞争力；Qwen3-0.6B 上达到比 GRPO 更好的安全性和相当推理表现，用时约为其七分之一。
正文 · 原文
Abstract:Large reasoning models achieve strong performance through reinforcement learning but can suffer safety degradation. External teacher distillation can restore safety yet may compromise reasoning performance. We formalize safety realignment as a KL projection: the student's native filter-accepted distribution is uniquely KL-optimal, while teacher targets incur excess KL when they differ. Guided by this analysis, we propose ThinkSafe, which generates safety-alignment data from the frozen student without external teachers. Refusal steering elicits safety reasoning, and a safety guard filters the responses before offline fine-tuning. Our analysis bounds steering-induced target distortion and gives a sufficient condition for lower average target KL than teacher distillation. Experiments on DeepSeek-R1-Distill and Qwen3 show substantial safety improvements with competitive reasoning performance. On Qwen3-0.6B, ThinkSafe achieves better safety and comparable reasoning to GRPO in about one-seventh of the wall-clock time, including data generation. Code, models, and datasets are available at this https URL and this https URL.
| Comments: | NeurIPS 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.23143 [cs.AI] |
| (or arXiv:2601.23143v5 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2601.23143 arXiv-issued DOI via DataCite |
Submission history
From: Seanie Lee [view email]
[v1]
Fri, 30 Jan 2026 16:31:02 UTC (605 KB)
[v2]
Fri, 8 May 2026 17:26:01 UTC (606 KB)
[v3]
Tue, 12 May 2026 15:49:44 UTC (597 KB)
[v4]
Wed, 13 May 2026 02:16:49 UTC (597 KB)
[v5]
Fri, 30 Oct 2026 10:17:31 UTC (1,218 KB)
来源：arXiv cs.AI · arxiv.org