UMER：基于成对感知判别推理统一多模态嵌入与排序
UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval
AI 导读
UMER 是一个统一多模态嵌入与排序的通用多模态检索框架，用 PairAware Discriminative Reasoning 取代逐条推理，通过比较查询-候选对识别指令相关的匹配与差异证据，并在单个 MLLM 内联合学习对比嵌入和判别式排序，辅以互相蒸馏策略。
正文 · 原文
Abstract:Modern Web information access increasingly involves heterogeneous multimodal content, including images, videos and visually rich documents. Users may seek not only globally similar content but also candidates that satisfy fine-grained attributes, relations or answer evidence under complex instructions. Supporting such access requires both efficient corpus-scale retrieval and fine-grained relevance reasoning. Multimodal large language model (MLLM)-based embedding methods derive representations from MLLM hidden states, and recent work further uses Chain-of-Thought (CoT) reasoning to improve embedding capability by injecting intermediate semantic evidence into the representation. However, existing CoT methods typically use item-wise reasoning over queries and candidates in isolation, providing no explicit evidence to distinguish a positive from a semantically confusable hard negative. Moreover, contrastive embeddings capture global similarity but struggle with meta-tasks requiring answer verification, category judgment or fine-grained reasoning. In this paper, we propose UMER, a Unified Multimodal Embedding and Ranking framework for universal multimodal retrieval. UMER replaces item-wise reflection with PairAware Discriminative Reasoning, which compares query-candidate pairs to identify instruction-relevant matching and discrepancy evidence. UMER jointly learns contrastive embeddings for efficient global matching and discriminative ranking for explicit pairwise relevance judgment within a single MLLM. A complementary mutual distillation strategy further transfers reliable pairwise preferences between the embedding and ranking functions. On MMEB-V2, UMER-R scores 65.5/70.6 with Qwen2-VL-2B/7B, outperforming the strongest included matched-backbone baselines by 1.4/2.0 points. UMER also supports budget-adjustable inference from efficient embedding retrieval to optional pair-aware reranking.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.18504 [cs.AI] |
| (or arXiv:2608.18504v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.18504 arXiv-issued DOI via DataCite |
Submission history
From: Xiyang Liu [view email]
[v1]
Wed, 19 Aug 2026 03:51:39 UTC (2,985 KB)
[v2]
Wed, 26 Aug 2026 06:46:11 UTC (2,985 KB)
[v3]
Wed, 11 Nov 2026 06:52:56 UTC (3,529 KB)
来源：arXiv cs.AI · arxiv.org