视觉语言模型 · CVPR2026 ·2026.06.03

RobustVisRAG:视觉退化下因果感知的基于视觉的检索增强生成

RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations

I-Hsiang Chen、Yu-Wei Liu、Tse-Yu Wu、Yu-Chien Chiang、Jen-Chieh Yang、Wei-Ting Chen

与你有关的部分

提出因果引导的双路径框架,通过非因果路径捕获退化信号并引导因果路径学习纯化语义,为视觉检索增强生成在退化条件下的鲁棒性提供了可复用的解耦思路;同时构建的 Distortion-VisRAG 数据集可用于评测多模态检索与生成模型在真实退化下的表现。

原文依据 RobustVisRAG uses a non-causal path to capture degradation signals through unidirectional attention and a causal path to learn purified semantics guided by these signals.
可迁移到 视觉文档检索多模态检索增强生成视觉问答图像退化鲁棒性

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

基于视觉的检索增强生成(VisRAG)利用视觉语言模型(VLM)联合检索相关视觉文档并基于多模态证据生成有依据的答案。然而,现有 VisRAG 模型在视觉输入遭受模糊、噪声、低光照或阴影等失真时性能下降,此时语义与退化因素在预训练视觉编码器中相互纠缠,导致检索和生成阶段均出现错误。为解决这一局限,我们提出 RobustVisRAG,一种因果引导的双路径框架,在保持效率和零样本泛化能力的同时提升 VisRAG 的鲁棒性。RobustVisRAG 使用非因果路径通过单向注意力捕获退化信号,并使用因果路径在这些信号引导下学习纯化语义。结合所提出的非因果失真建模和因果语义对齐目标,该框架强制语义与退化之间清晰分离,从而在挑战性视觉条件下实现稳定的检索和生成。为评估真实条件下的鲁棒性,我们引入 Distortion-VisRAG 数据集,这是一个大规模基准,包含跨越七个领域的合成和真实退化文档,涵盖 12 种合成和 5 种真实失真类型,全面反映实际视觉退化。实验结果表明,RobustVisRAG 在真实退化下分别将检索、生成和端到端性能提升 7.35%、6.35% 和 12.40%,同时在干净输入上保持相当的准确率。

英文原文摘要 1636 字符

Vision-based Retrieval-Augmented Generation (VisRAG) leverages vision-language models (VLMs) to jointly retrieve relevant visual documents and generate grounded answers based on multimodal evidence. However, existing VisRAG models degrade in performance when visual inputs suffer from distortions such as blur, noise, low light, or shadow, where semantic and degradation factors become entangled within pretrained visual encoders, leading to errors in both retrieval and generation stages. To address this limitation, we introduce RobustVisRAG, a causality-guided dual-path framework that improves VisRAG robustness while preserving efficiency and zero-shot generalization. RobustVisRAG uses a non-causal path to capture degradation signals through unidirectional attention and a causal path to learn purified semantics guided by these signals. Together with the proposed Non-Causal Distortion Modeling and Causal Semantic Alignment objectives, the framework enforces a clear separation between semantics and degradations, enabling stable retrieval and generation under challenging visual conditions. To evaluate robustness under realistic conditions, we introduce the Distortion-VisRAG dataset, a large-scale benchmark containing both synthetic and real-world degraded documents across seven domains, with 12 synthetic and 5 real distortion types that comprehensively reflect practical visual degradations. Experimental results show that RobustVisRAG improves retrieval, generation, and end-to-end performance by 7.35%, 6.35%, and 12.40%, respectively, on real-world degradations, while maintaining comparable accuracy on clean inputs.

标签

会议 CVPR2026
页码 40802-40811
Bibkey
来源 CVF Open Access