更好的 Reward 模型不一定更少偏见：超越标准 Reward 基准的社会对齐研究
Better Reward Models Are Not Always Less Biased: Social Alignment Beyond Standard Reward Benchmarks
AI 导读
一项 arXiv 研究将 reward 模型的评测扩展到偏见、安全、道德与伦理推理四个社会性领域，发现标准基准上的强劲表现并不能保证社会对齐行为。分析显示，模型可能回避某些刻板印象的同时强化另一些，聚合偏见分数会掩盖这些相反倾向；更强的刻板印象回避还会牺牲对上下文的忠实度。研究还发现基准表现与社会对齐行为仅不一致地相关，呼吁对 reward 模型编码的社会偏好做更直接的评测。
正文 · 原文
Abstract:Reward models are central to LLM alignment, yet current benchmarks reveal little about how they represent diverse and sometimes competing social preferences. We extend reward model evaluation to four socially consequential domains: bias, safety, morality, and ethical reasoning. Our analysis shows that strong benchmark performance can obscure important differences in social behavior. Models may avoid some stereotypes while reinforcing others, and aggregate bias scores can hide these opposing tendencies. We also find that stronger stereotype avoidance can come at the expense of contextual faithfulness, revealing a trade-off between social bias mitigation and sensitivity to context. Across a broad set of reward models, standard benchmark performance is only inconsistently related to socially aligned behavior. These results show that stronger reward models are not uniformly better at navigating pluralistic social expectations and motivate more direct evaluation of the social preferences encoded in reward models.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2605.05003 [cs.CL] |
| (or arXiv:2605.05003v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.05003 arXiv-issued DOI via DataCite |
Submission history
From: Esra Dönmez [view email]
[v1]
Wed, 6 May 2026 15:04:23 UTC (8,599 KB)
[v2]
Fri, 23 Oct 2026 15:12:30 UTC (6,152 KB)
来源：arXiv cs.AI · arxiv.org