Suan：修正大语言模型的直接偏好安全对齐
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
AI 导读
arXiv 论文提出 Suan，一种新的偏好优化算法，用于改进大语言模型的安全对齐。该算法直接在梯度层面构建优化目标，绕过标准的变分推导，获得更可解释、更稳健的训练动态，在多组基线与基准评测中实现更强的安全对齐，同时完整保留响应质量。
正文 · 原文
Abstract:Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.08634 [cs.LG] |
| (or arXiv:2609.08634v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.08634 arXiv-issued DOI via DataCite |
Submission history
From: Roman Klypa [view email]
[v1]
Tue, 8 Sep 2026 12:04:38 UTC (426 KB)
[v2]
Fri, 2 Oct 2026 10:27:05 UTC (426 KB)
[v3]
Mon, 26 Oct 2026 19:28:50 UTC (426 KB)
来源：arXiv cs.AI · arxiv.org