arXiv cs.AI· Tianyu Yang, Yijun Tian, Lars Benedikt Kaesberg, Jan Philip Wahle, Bela Gipp, Terry Ruas, Daniel Kurzawe·· 2 小时前AI 评分36
PILOT：面向多模态嵌入的特权推理探索框架
Beyond Post-Hoc Rewards: Privileged Reasoning Exploration for Multimodal Embeddings
AI 导读
论文提出 PILOT，将多模态嵌入中的自生成推理建模为隐变量，用仅依赖输入的先验与利用训练期关系证据的后验两种推理模式进行协同优化，并选择性把后验轨迹迁移给先验。在 78 项任务的 MMEB-V2 基准上，PILOT 在 2B 和 7B 参数下均超越主要基线，总分分别提升 3.2 和 3.3 点，视觉文档检索增益最大（+6.1 和 +6.4 点）。
正文 · 原文
Abstract:Advances in Multimodal Large Language Models (MLLMs) have driven substantial progress in multimodal embeddings. Recent methods exploit generative capabilities by augmenting embedded inputs with self-generated reasoning to produce more informative embeddings. However, conventional retrieval supervision for training embedding models specifies pairwise relevance relations without explicitly guiding the reasoning used for input augmentation. Such reasoning is neither directly observed nor uniquely determined by the input, allowing multiple plausible traces to introduce semantic cues of varying retrieval utility. Existing approaches largely optimize reasoning conditioned on the input alone, limiting the exploration of retrieval-relevant traces. We introduce PILOT, a framework that models reasoning as a latent variable and instantiates its prior and posterior distributions as distinct reasoning modes of a shared MLLM. The prior conditions only on the input, whereas the posterior uses additional training-time relational evidence to guide reasoning exploration. To translate the posterior's broader exploration into inference-time improvements, we derive an importance-weighted variational objective and instantiate it as a prior-posterior co-evolution framework that improves both modes under a shared inference-compatible reward while selectively transferring posterior trajectories to the prior according to their utility and prior reachability. On the 78-task MMEB-V2 benchmark, PILOT outperforms the primary baseline across all 12 task groups at both 2B and 7B parameters, improving overall scores by 3.2 and 3.3 points, respectively, with the largest gains in visual-document retrieval (+6.1 and +6.4 points). Our results show that relational supervision is most valuable not as a post-hoc reward but as a signal for shaping how reasoning is explored in training reasoning-enhanced embedding models.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.32474 [cs.AI] |
| (or arXiv:2610.32474v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32474 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianyu Yang [view email]
[v1]
Fri, 23 Oct 2026 10:43:41 UTC (640 KB)
来源：arXiv cs.AI · arxiv.org