面向含噪偏好数据的无偏对齐：URM 与 UDPO 损失框架
Unbiased Alignment for Large Language Models with Noisy Preferences
AI 导读
一项 arXiv 论文提出无偏对齐理论框架，引入 Unbiased Reward Model（URM）损失和 Unbiased Direct Preference Optimization（UDPO）损失，数学上修正偏好噪声带来的偏差，使模型可直接在含噪偏好数据集上训练，无需干净的真实标注。
正文 · 原文
Abstract:The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: this https URL.
| Comments: | Accepted by ICML 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.03248 [cs.LG] |
| (or arXiv:2607.03248v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.03248 arXiv-issued DOI via DataCite |
Submission history
From: Jialiang Wang [view email]
[v1]
Fri, 3 Jul 2026 12:04:43 UTC (1,699 KB)
[v2]
Mon, 26 Oct 2026 15:40:42 UTC (1,700 KB)
来源：arXiv cs.AI · arxiv.org