跳到正文
arXiv cs.AI· Xingzi Xu, Mehmet Saygin Seyfioglu, Karim Bouyarmane·· 56 分钟前AI 评分50

arXiv 论文提出边界动力学理论,解释 DPO 与 IPO 对齐行为并推导 SimPO 有限吸引子

Margin Dynamics for Large Language Model Alignment

AI 导读

论文为 LLM 对齐方法提供理论基础,将成对方法归结为一维边界 ODE 并给出闭式解,证明 DPO 边界按 O(log t) 无界增长而 IPO 收敛到 1/(2τ),标签噪声使 DPO 发生从无界到有界的相变,结论在五个模型家族、四个基准上验证。

正文 · 原文

View PDF HTML (experimental)

Abstract:Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as $O(\log t)$ in the gradient-flow limit) while IPO margins converge to a finite $1/(2\tau)$; that label noise induces a qualitative phase transition in DPO (unbounded $\to$ bounded) but affects IPO only quantitatively; and that gradient concentration under asymmetric weighting governs trainability at scale. We confirm all three predictions across five model families spanning four benchmarks. The closed-form solutions also enable principled method design: we prove that adding any label smoothing $\varepsilon > 0$ to SimPO creates a finite margin attractor at $\Delta^* = \beta^{-1}\log((1-\varepsilon)/\varepsilon)$, while $\varepsilon = 0$ (standard SimPO) has no attractor. We test this across three models (8B-9B) and confirm the prediction: the theory-derived configuration (TPO-S, $\varepsilon = 0.05$) is competitive with the strongest baselines, attaining the highest mean IFEval score among the compared methods on all three, validating that the predicted finite attractor is a usable operating point rather than proposing a new state-of-the-art loss.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.21766 [cs.LG]
  (or arXiv:2611.21766v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.21766

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xingzi Xu PhD [view email]
[v1] Mon, 16 Nov 2026 19:02:17 UTC (96 KB)

来源:arXiv cs.AI · arxiv.org