ReAG：面向知识型视觉问答的推理增强生成
ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
对做视觉语言模型的人，这篇给出了一个可复用的多模态 RAG 流程：粗粒度+细粒度两阶段检索、critic 模型过滤噪声段落、再用强化学习让模型对检索内容做推理，可直接迁移到需要外部知识的 VQA 或 captioning 场景。
原文依据 combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）在联合理解文本、图像和视频方面展现出令人印象深刻的能力，通常通过视觉问答（VQA）进行评估。然而，即便是最优的 MLLMs 也难以应对领域特定或知识密集型的查询，因为相关信息在预训练数据中代表性不足。知识型视觉问答（KB-VQA）通过检索外部文档来为答案生成提供条件，从而应对这一问题，但当前的检索增强方法存在精度低、段落噪声大、推理能力有限等问题。为解决这些问题，我们提出 ReAG，一种新颖的推理增强多模态 RAG 方法，它将粗粒度与细粒度检索与一个过滤无关段落的 critic 模型相结合，确保高质量的额外上下文。该模型采用多阶段训练策略，利用强化学习增强对检索内容的推理，而监督微调仅作为冷启动。在 Encyclopedic-VQA 和 InfoSeek 上的大量实验表明，ReAG 显著优于先前方法，提升了答案准确率，并提供了基于检索证据的可解释推理。
英文原文摘要 1197 字符
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in jointly understanding text, images, and videos, often evaluated via Visual Question Answering (VQA). However, even state-of-the-art MLLMs struggle with domain-specific or knowledge-intensive queries, where relevant information is underrepresented in pre-training data. Knowledge-based VQA (KB-VQA) addresses this by retrieving external documents to condition answer generation, but current retrieval-augmented approaches suffer from low precision, noisy passages, and limited reasoning. To address this, we propose ReAG, a novel Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages, ensuring high-quality additional context. The model follows a multi-stage training strategy leveraging reinforcement learning to enhance reasoning over retrieved content, while supervised fine-tuning serves only as a cold start. Extensive experiments on Encyclopedic-VQA and InfoSeek demonstrate that ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.