arXiv cs.AI· Erhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu, Francesco Quinzan, Chengchun Shi·· 4 小时前AI 评分39
面向大语言模型的双重鲁棒对齐算法研究
Doubly Robust Alignment for Large Language Models
AI 导读
论文针对 RLHF 中偏好模型（如 Bradley-Terry 模型）、参考策略或奖励函数被错误设定导致微调失效的问题，提出一种双重鲁棒的偏好优化算法。该算法在偏好模型或参考策略二者其一正确设定时即可保持一致性，无需两者同时正确。理论与实验均显示其性能优于 SOTA 算法且更稳健，代码已开源，论文被 NeurIPS 2025 接收。
正文 · 原文
Abstract:This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at this https URL
| Comments: | Accepted to NeurIPS 2025 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2506.01183 [cs.LG] |
| (or arXiv:2506.01183v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.01183 arXiv-issued DOI via DataCite |
Submission history
From: Erhan Xu [view email]
[v1]
Sun, 1 Jun 2025 21:34:37 UTC (973 KB)
[v2]
Tue, 28 Oct 2025 20:58:26 UTC (1,018 KB)
[v3]
Tue, 27 Oct 2026 08:23:56 UTC (1,003 KB)
来源：arXiv cs.AI · arxiv.org