重新思考文本到图像奖励建模中的视觉推理
Rethinking Visual Reasoning in Text-to-Image Reward Modeling
AI 导读
一项 arXiv 研究发现,主流文本到图像奖励模型在针对性视觉推理评测中表现不佳,偏好微调甚至会削弱视觉语言骨干的属性绑定、关系解析和计数能力。研究者提出将现有视觉推理数据集转换为共享同一图像、提示词不同的偏好对,与人类偏好数据混合训练,实现视觉理解与偏好预测的 Pareto 改进。
正文 · 原文
Abstract:Reward models are a core component of modern text-to-image systems, serving as the objective for inference-time selection and RL-based post-training. A key prerequisite is visual understanding: to judge prompt adherence and ground aesthetic preferences, a reward model must bind attributes, resolve relations, and count reliably. We find that widely-used reward models perform poorly on targeted visual reasoning evaluations. Even when starting from strong vision-language backbones, preference fine-tuning erodes these capabilities, and continual-learning baselines fail to mitigate a steep tradeoff between visual reasoning and preference accuracy. We propose a data-centric intervention that injects visual reasoning supervision into preference learning by flipping the pairing axis. To complement conventional human preference data (paired images under a shared prompt), we convert existing visual reasoning datasets into preference pairs that share the same image but differ in prompt, and train on the mixture. This yields a Pareto improvement in both visual understanding and preference prediction. Across inference-time best-of-n selection and RL post-training, the resulting reward model improves text-to-image generation on challenging compositional prompts (GenEval) by an average of 4.0% and 5.5% respectively, outperforming HPSv3 which was trained on 6x more preference pairs.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.17585 [cs.CV] |
| (or arXiv:2611.17585v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.17585 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shiye Su [view email]
[v1]
Thu, 12 Nov 2026 23:09:28 UTC (13,136 KB)
来源:arXiv cs.AI · arxiv.org