MR-RAG：面向医学视觉问答的多模态相关性感知检索增强生成
MR-RAG: Multimodal Relevance-Aware Retrieval-Augmented Generation for Medical Visual Question Answering
对做医学 VQA 或多模态 RAG 的研究者，这篇给出了一个可复用的双阶段思路：检索端用模态内与跨模态信号联合打分做文档对齐，生成端按融合后的相关性重加权注意力路径，可直接迁移到其他需要外部知识注入的视觉问答系统。
原文依据 an Importance-Aware Information Flow Augmentation (IFA) mechanism adjusts attention paths based on fused relevance to refine answer generation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论具备检索增强生成（RAG）能力的大视觉语言模型（LVLM）凭借其可观的成果，正成为处理视觉-语言医学任务的主流范式。然而，现有方法在检索阶段与生成阶段均存在两个显著局限：第一，在检索阶段，多数方法通常依赖单一相似度信号来估计文档相关性，忽略了多模态数据中蕴含的丰富信息，可能无法准确检索到匹配内容。第二，在生成阶段，检索到的文档被直接、统一地拼接进 LVLM 的输入，而未考虑它们与问题的相关性差异，这可能稀释关键信息并加剧无关内容的负面影响。为解决上述局限，我们提出 MR-RAG，一个在检索与生成两个阶段均考虑多模态相关性的双阶段 RAG 增强框架。具体而言，多模态协同检索（MCR）模块利用模态内与跨模态信号实现精确的文档对齐。随后，重要性感知信息流增强（IFA）机制依据融合后的相关性调整注意力路径，以精化答案生成。通过多模态信号打通检索与生成，MR-RAG 显著提升了事实准确性。在三个医学数据集上的实验表明，本方法优于最优基线，准确率最高提升 6.4%。
英文原文摘要 1537 字符
Large Vision Language Models (LVLMs) with retrieval-augmented generation (RAG) are emerging as a main paradigm for processing vision-language medical tasks due to their promising achievements. However, existing approaches exhibit two significant limitations in both retrieval and generation stage: First, during the retrieval stage, most methods typically rely on a single similarity signal to estimate document relevance, ignoring the rich information available in multimodal data, which may fail to accurately retrieve matching content. Second, in the generation stage, retrieved documents are integrated directly and uniformly into the input for LVLMs, without taking into account their varying relevance to the question, which may result in the dilution of crucial information and exacerbate the negative impact of irrelevant content. To address these limitations, we propose MR-RAG, a dual-stage RAG enhancement framework by considering multimodal relevance in both retrieval and generation phases. Specifically, a Multimodal Cooperative Retrieval (MCR) module leverages intra- and cross-modal signals for precise document alignment. Subsequently, an Importance-Aware Information Flow Augmentation (IFA) mechanism adjusts attention paths based on fused relevance to refine answer generation. By bridging retrieval and generation via multimodal signals, MR-RAG significantly improves factual accuracy. Experiments on three medical datasets show our method outperforms state-of-the-art baselines, achieving up to a 6.4% accuracy gain.