OGC-DPO：面向多目标对齐的目标条件化主动偏好学习
Objective-Conditioned Active Preference Learning for Multi-Objective Alignment
AI 导读
研究提出 OGC-DPO（Objective-conditioned Gradient Coverage DPO），一种用于 LLM 多目标对齐（如有用性、无害性、诚实性）的主动偏好学习框架。
正文 · 原文
Abstract:Aligning large language models (LLMs) with several objectives, such as helpfulness, harmlessness, and honesty, typically relies on preference judgment for each objective, growing the annotation cost linearly with the number of objectives. Active preference learning can reduce this cost by selecting the comparisons whose labels are expected to change the policy most. Existing remedies either lack an objective-aware query selection mechanism for the multi-objective setting or regard conflicting preferences as harmful, therefore excluding them from the training data. We introduce OGC-DPO (Objective-conditioned Gradient Coverage DPO), an active learning framework that treats each (response pair, objective) combination as a separate query and measures its informativeness through the policy's own implicit gradient, without relying on an external reward model. OGC-DPO then selects the queries that most expand each objective's gradient coverage. Empirical experiments across varying numbers of objectives demonstrate the effectiveness of OGC-DPO over existing active preference learning methods under the same annotation cost. Further analysis demonstrates that informative conflicts, rather than being uniformly harmful, provide a useful policy training signal.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.05506 [cs.LG] |
| (or arXiv:2611.05506v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05506 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Duc Anh Vu [view email]
[v1]
Tue, 3 Nov 2026 19:04:16 UTC (213 KB)
来源：arXiv cs.AI · arxiv.org