arXiv cs.AI· Yuhang Han, Dingning Liu, Yujie Chen, Yaojie Zhang, Jingwei Song, Zequn Yang, Xuanming Shang, Li Ding, Zhijiang Guo, Linfeng Zhang·· 2 小时前AI 评分33
OVC：从视觉亲和到学习信用分配的多模态强化学习方法
From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning
AI 导读
一项研究提出 Outcome-Conditioned Visual Credit（OVC），将视觉关联度量与 token 级信用分配分离，以改进多模态推理中的 RLVR 训练。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but uniformly assigning a single response-level advantage to all tokens overlooks the distinct roles of visual observations and subsequent reasoning. Visual affinity alone is insufficient: predictive states may align with both image and prompt-context representations, and upweighting valid visual observations can amplify their negative-advantage updates when later reasoning fails. To address this issue, we propose \textbf{Outcome-Conditioned Visual Credit (OVC)}, a plug-in framework that separates visual-association measurement from token-level credit assignment and is adaptable to policy-optimization methods with uniform response-level advantage allocation. Relative Visual Encoding constructs a context-relative visual coordinate by contrasting each predictive state's affinity to image and prompt-context representations. Outcome-conditioned Evidence Calibration combines this coordinate with uncertainty-scaled predictive features and calibrates their contributions to outcome-discriminative evidence through mean contrasts between positive- and negative-advantage responses, scaled by regularized within-side variation. Visual association thus informs credit through observed outcome differences rather than directly determining update priority. Normalized Credit Allocation conditions the calibrated evidence on the response's advantage sign to produce positive token weights with unit global mean. These detached weights preserve advantage signs while modulating individual token-gradient contributions. OVC reuses existing rollouts without auxiliary reward models, process annotations, or inference-time modules. Instantiated through GSPO-token, OVC retains sequence-level clipping and consistently improves six-benchmark mean accuracy over GSPO across three models, with gains of 1.86-2.39 percentage points.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32324 [cs.AI] |
| (or arXiv:2610.32324v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32324 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yujie Chen [view email]
[v1]
Fri, 23 Oct 2026 08:22:16 UTC (527 KB)
来源：arXiv cs.AI · arxiv.org