arXiv cs.AI· Jinwoo Kim, Rina Carines Cabral, Dhita Putri Pratama, Eileen Wang, Hiba Arnaout, Shuo Yang, Danyang Liu, Seungyeon Ji, Jie Yang, Josiah Poon, Jeff Z. Pan, Kyungreem Han, Soyeon Caren Han·· 3 小时前AI 评分43
MMCOMET：面向视觉推理的大规模多模态常识知识图谱
MMCOMET: A Large-Scale Multimodal Commonsense Knowledge Graph for Visual Reasoning
AI 导读
研究者推出 MMCOMET，一个将物理、社会与事件知识结合视觉 grounding 的大规模多模态常识知识图谱。它通过混合对齐流程扩展 ATOMIC2020，为每条常识三元组关联代表性图像，形成超 900K 多模态元组、覆盖 19 种关系类型，并建立了涵盖视觉常识推理、视觉问答等任务的统一评测框架。
正文 · 原文
Authors:Jinwoo Kim, Rina Carines Cabral, Dhita Putri Pratama, Eileen Wang, Hiba Arnaout, Shuo Yang, Danyang Liu, Seungyeon Ji, Jie Yang, Josiah Poon, Jeff Z. Pan, Kyungreem Han, Soyeon Caren Han
Abstract:Commonsense knowledge is widely used to support multimodal reasoning and generation, yet existing resources are predominantly text-only and lack grounding in visual context. This limitation often leads to the retrieval of semantically related but visually misaligned knowledge, and can degrade downstream performance. We introduce MMCOMET, a large-scale multimodal commonsense knowledge graph that integrates physical, social, and eventive knowledge with visual grounding. MMCOMET extends ATOMIC2020 by associating each commonsense triple with representative images through a hybrid alignment pipeline, resulting in over 900K multimodal tuples spanning 19 relation types. We further establish a unified evaluation framework across diverse vision-language tasks, including visual commonsense reasoning, visual question answering, visual storytelling, image editing, and image captioning. Our analysis shows that multimodal grounding provides more contextually aligned and reliable knowledge compared to both text-only knowledge graphs and LLM-based knowledge generation, particularly in reasoning-intensive scenarios. At the same time, we observe that the effectiveness of external knowledge is task-dependent, and naive knowledge integration can introduce noise in perception-dominated settings. To mitigate this, we propose an adaptive knowledge router that dynamically selects optimal paths or bypass knowledge in harmful settings. MMCOMET provides a scalable multimodal commonsense resource and a benchmark for studying the role of grounded knowledge in multimodal systems.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.01055 [cs.AI] |
| (or arXiv:2603.01055v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2603.01055 arXiv-issued DOI via DataCite |
Submission history
From: Jinwoo Kim [view email]
[v1]
Sun, 1 Mar 2026 11:21:50 UTC (7,620 KB)
[v2]
Mon, 2 Nov 2026 04:28:17 UTC (48,037 KB)
[v3]
Tue, 10 Nov 2026 04:13:56 UTC (48,037 KB)
来源：arXiv cs.AI · arxiv.org