跳到正文
arXiv cs.AI· Zongyu Wu, Yilong Wang, Xiaochen Wang, Minhua Lin, Zhichao Xu, Fenglong Ma, Xiang Zhang, Suhang Wang·· 2 小时前AI 评分29

CEMMKG:多粒度上下文增强的多模态知识图谱 RAG 框架

Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphs

AI 导读

研究者提出 CEMMKG 框架,通过在局部和全局两个范围上为每张图像补充文本上下文来构建上下文增强多模态知识图谱,以缓解图像与文本间的语义鸿沟。局部上下文采用多粒度设计,融合与图像语义相关的句子而非仅限邻近文本。多数据集实验显示 CEMMKG 能提升多模态 GraphRAG 性能,并适用于多种基于 MMKG 的 RAG 方法。

正文 · 原文

View PDF HTML (experimental)

Abstract:Knowledge graph (KG)-based retrieval-augmented generation (RAG) leverages structured knowledge to provide large language models (LLMs) with high-quality external information. Building on these works, recent studies have explored multimodal knowledge graphs (MMKGs) as knowledge bases for GraphRAG. This enables GraphRAG to integrate knowledge across multiple modalities, thereby further enhancing its performance. However, existing MMKG-based RAG methods generally follow a common pipeline in which different modalities are largely processed independently before being fused. As a result, textual context is only used to a limited extent during visual information extraction and subsequent multimodal knowledge fusion. This brings a semantic gap between images and text which limits the multimodal GraphRAG performance. To address this issue, we propose a novel framework for constructing a Context-Enhanced MMKG (CEMMKG) to better support multimodal GraphRAG. The proposed CEMMKG enriches each image with complementary textual context at both local and global scopes. Local context goes beyond the surrounding text by incorporating sentences that are semantically related to the image, while global context provides a summary of the entire document. We further introduce a multi-granularity design for the local context, allowing it to capture semantically relevant information at different levels of detail. Extensive experiments on multiple datasets validate that CEMMKG is effective in leveraging contextual information to improve MMKG-based RAG performance. Moreover, its effectiveness across different MMKG-based RAG methods demonstrates its broad applicability.
Comments: Preprint
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2608.25986 [cs.AI]
  (or arXiv:2608.25986v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2608.25986

arXiv-issued DOI via DataCite

Submission history

From: Zongyu Wu [view email]
[v1] Wed, 26 Aug 2026 16:38:02 UTC (553 KB)
[v2] Tue, 17 Nov 2026 11:26:05 UTC (311 KB)

来源:arXiv cs.AI · arxiv.org