arXiv cs.AI· Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin, Bryan Kian Hsiang Low·· 4 小时前AI 评分35
UMPIRE:无需训练的多模态大语言模型不确定性量化框架
Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume
AI 导读
研究者提出 UMPIRE,一种无需训练的多模态大语言模型(MLLM)不确定性量化框架,不依赖外部工具,仅用模型自身内部模态特征。该框架受 Determinantal Point Processes 启发,通过计算"非相干调整语义体积"捕捉样本全局语义多样性与响应局部非相干性。实验显示其在图像、音频、视频文本基准上的错误检测与不确定性校准均优于基线,且可泛化到图像和音频生成任务,代码已开源。
正文 · 原文
Abstract:Despite their capabilities, Multimodal Large Language Models (MLLMs) may produce plausible but erroneous outputs, hindering reliable deployment. Accurate uncertainty metrics could enable escalation of unreliable queries to human experts or larger models for improved performance. However, existing uncertainty metrics have practical constraints, such as being designed only for specific modalities, reliant on external tools, or computationally expensive. We introduce UMPIRE, a training-free uncertainty quantification framework for MLLMs that works efficiently across various input and output modalities without external tools, relying only on the models' own internal modality features. Inspired by Determinantal Point Processes (DPPs), UMPIRE computes a novel incoherence-adjusted semantic volume of sampled MLLM responses for a given task instance, which couples semantic embedding dispersion with the quadratic entropy of the responses. This approach effectively captures both the global semantic diversity of samples and the local incoherence of responses based on internal model confidence. We provide theoretical analysis motivating UMPIRE's design, and extensive experiments show that UMPIRE consistently outperforms baseline metrics in error detection and uncertainty calibration across image, audio, and video-text benchmarks, even with small sample budgets. We also demonstrate UMPIRE's generalization to non-text output tasks, including image and audio generation. Our code is available at this https URL.
| Comments: | Accepted at NeurIPS 2026. Earlier versions presented at ICLR 2025 QUESTION workshop and ICML 2025 R2-FM workshop |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2602.24195 [cs.AI] |
| (or arXiv:2602.24195v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2602.24195 arXiv-issued DOI via DataCite |
Submission history
From: Hieu Dao [view email]
[v1]
Fri, 27 Feb 2026 17:18:42 UTC (3,783 KB)
[v2]
Fri, 13 Nov 2026 08:41:34 UTC (3,824 KB)
来源:arXiv cs.AI · arxiv.org