M^3KG-RAG：多跳多模态知识图谱增强的检索增强生成
M^3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
对做多模态大模型视觉问答与视觉定位的研究者，这篇提供了可复用的思路：用多智能体流水线自动构建多跳多模态知识图谱，并用 GRASP 做实体视觉定位与相关性评估后剪枝冗余上下文，可直接迁移到视频/音视频问答的检索增强生成流程中，缓解仅靠嵌入相似度检索带来的离题噪声。
原文依据 we introduce GRASP (Grounded Retrieval And Selective Pruning), which ensures precise entity grounding to the query
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论检索增强生成（RAG）近来被扩展到多模态场景，将多模态大语言模型（MLLM）与多模态知识图谱（MMKG）等大规模外部知识语料相连接。尽管近期取得了成功，音视频领域的多模态 RAG 仍面临两方面挑战：1）现有 MMKG 的模态覆盖范围与多跳连通性有限；2）仅基于共享多模态嵌入空间中的相似度进行检索，无法过滤掉离题或冗余的知识。为解决这些局限，我们提出 M^3KG-RAG，一种多跳多模态知识图谱增强的 RAG，它从 MMKG 中检索与查询对齐的音视频知识，从而提升 MLLM 的推理深度与答案忠实度。具体而言，我们设计了一个轻量级多智能体流水线来构建多跳 MMKG（M^3KG），其中包含上下文丰富的多模态实体三元组，使得能够基于输入查询进行按模态的检索。此外，我们引入 GRASP（视觉定位检索与选择性剪枝），它确保实体与查询的精确视觉定位，评估支撑答案的相关性，并剪除冗余上下文，仅保留对回答生成至关重要的知识。在多种多模态基准上的大量实验表明，M^3KG-RAG 相比现有方法显著增强了 MLLM 的多模态推理与视觉定位能力。
英文原文摘要 1422 字符
Retrieval-Augmented Generation (RAG) has recently been extended to multimodal settings, connecting multimodal large language models (MLLMs) with vast corpora of external knowledge such as multimodal knowledge graphs (MMKGs). Despite their recent success, multimodal RAG in the audio-visual domain remains challenging due to 1) limited modality coverage and multi-hop connectivity of existing MMKGs, and 2) retrieval based solely on similarity in a shared multimodal embedding space, which fails to filter out off-topic or redundant knowledge. To address these limitations, we propose M^3KG-RAG, a Multi-hop Multimodal Knowledge Graph-enhanced RAG that retrieves query-aligned audio-visual knowledge from MMKGs, improving reasoning depth and answer faithfulness in MLLMs. Specifically, we devise a lightweight multi-agent pipeline to construct multi-hop MMKG (M^3KG), which contains context-enriched triplets of multimodal entities, enabling modality-wise retrieval based on input queries. Furthermore, we introduce GRASP (Grounded Retrieval And Selective Pruning), which ensures precise entity grounding to the query, evaluates answer-supporting relevance, and prunes redundant context to retain only knowledge essential for response generation. Extensive experiments across diverse multimodal benchmarks demonstrate that M^3KG-RAG significantly enhances MLLMs' multimodal reasoning and grounding over existing approaches.