GAP：面向多模态大语言模型视觉推理的细粒度对齐范式
Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models
研究者提出视觉潜在推理对齐范式 GAP，解决现有"输出即输入"潜在范式增益不稳定的问题，将其归因于 decoder 隐状态与输入 embedding 范数区间的特征空间错配。GAP 在数据级、特征级和模型级三个层面进行对齐，包括轻量 PCA 对齐潜在头。在 Qwen2.5-VL 7B 上，GAP 取得所有对比方法中最优的感知与推理总分，且是唯一同时优于基座的潜在推理方法，代码已开源。
Authors:Yanting Miao, Yutao Sun, Dexin Wang, Mengyu Zhou, Pascal Poupart, Lei Lv, Qi Zhao, Li Wang, Hao Li, Xiaoxi Jiang, Guanjun Jiang
Abstract:Visual latent reasoning lets a multimodal large language model (MLLM) create intermediate visual evidence as continuous tokens, avoiding external tools or image generators. However, existing methods usually follow an output-as-input latent paradigm and yield unstable gains. We identify a feature-space mismatch behind this instability: dominant visual-latent models build on pre-norm MLLMs and reuse decoder hidden states as predicted latent inputs, even though these states occupy a substantially different norm regime from the input embeddings the model was trained to consume. This mismatch makes direct latent feedback unreliable. Motivated by this diagnosis, we propose GAP, a Granular Alignment Paradigm for visual latent modeling. GAP aligns visual latent reasoning at three levels: data-level alignment grounds latent targets with inspectable auxiliary visual supervision; feature-level alignment maps decoder outputs into input-compatible visual latents through a lightweight PCA-aligned latent head; and model-level alignment assigns latent supervision selectively to examples where the base MLLM struggles. On Qwen2.5-VL 7B, GAP achieves the best aggregate perception and reasoning scores among all compared methods and is the only evaluated latent method that improves both over the base model. Inference-time interventions further show that generated latents carry task-relevant visual signal rather than merely adding token slots. Code is available at this https URL.
| Comments: | Accepted at NeurIPS 2026. 26 pages |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.12374 [cs.CV] |
| (or arXiv:2605.12374v5 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.12374 arXiv-issued DOI via DataCite |
Submission history
From: Yanting Miao [view email]
[v1]
Tue, 12 May 2026 16:41:09 UTC (1,444 KB)
[v2]
Fri, 15 May 2026 06:51:05 UTC (1,444 KB)
[v3]
Wed, 20 May 2026 07:37:13 UTC (1,444 KB)
[v4]
Mon, 25 May 2026 11:34:51 UTC (1,444 KB)
[v5]
Wed, 11 Nov 2026 03:21:08 UTC (1,301 KB)
来源：arXiv cs.AI · arxiv.org