arXiv 论文提出边界动力学理论,解释 DPO 与 IPO 对齐行为并推导 SimPO 有限吸引子
Margin Dynamics for Large Language Model Alignment
AI 导读
论文为 LLM 对齐方法提供理论基础,将成对方法归结为一维边界 ODE 并给出闭式解,证明 DPO 边界按 O(log t) 无界增长而 IPO 收敛到 1/(2τ),标签噪声使 DPO 发生从无界到有界的相变,结论在五个模型家族、四个基准上验证。
正文 · 原文
Abstract:Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as $O(\log t)$ in the gradient-flow limit) while IPO margins converge to a finite $1/(2\tau)$; that label noise induces a qualitative phase transition in DPO (unbounded $\to$ bounded) but affects IPO only quantitatively; and that gradient concentration under asymmetric weighting governs trainability at scale. We confirm all three predictions across five model families spanning four benchmarks. The closed-form solutions also enable principled method design: we prove that adding any label smoothing $\varepsilon > 0$ to SimPO creates a finite margin attractor at $\Delta^* = \beta^{-1}\log((1-\varepsilon)/\varepsilon)$, while $\varepsilon = 0$ (standard SimPO) has no attractor. We test this across three models (8B-9B) and confirm the prediction: the theory-derived configuration (TPO-S, $\varepsilon = 0.05$) is competitive with the strongest baselines, attaining the highest mean IFEval score among the compared methods on all three, validating that the predicted finite attractor is a usable operating point rather than proposing a new state-of-the-art loss.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.21766 [cs.LG] |
| (or arXiv:2611.21766v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21766 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xingzi Xu PhD [view email]
[v1]
Mon, 16 Nov 2026 19:02:17 UTC (96 KB)
来源:arXiv cs.AI · arxiv.org