通过自批判推理框架扩展视觉语言模型的测试时鲁棒性
Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework
提出自批判推理框架，通过多轮文本与视觉扰动的反事实推理提升LVLM鲁棒性，并引入动态鲁棒性基准DRBench，为视觉语言模型的可靠性评估与测试时增强提供可复用方法。
原文依据 we propose a novel Self-Critical Inference (SCI) framework that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大语言模型（LLMs）的出现推动了多模态学习的快速进展，尤其是大型视觉语言模型（LVLMs）的发展。然而，现有的LVLM训练范式过度依赖LLM组件，引发了语言偏差和语言敏感性这两个关键的鲁棒性挑战。为了同时解决这两个问题，我们提出了一种新颖的自批判推理（SCI）框架，该框架通过文本和视觉扰动进行多轮反事实推理，扩展了视觉对比解码。这一过程进一步引入了一种通过扩展反事实轮数来提高鲁棒性的新策略。此外，我们还观察到LVLMs的失败案例在不同模型间存在显著差异，这表明固定的鲁棒性基准可能无法捕捉LVLMs的真实可靠性。为此，我们提出了动态鲁棒性基准（DRBench），一个针对语言偏差和敏感性问题的模型特定评估框架。大量实验表明，SCI在DRBench上始终优于基线方法，并且增加推理轮数能进一步提升鲁棒性，超越现有的单步反事实推理方法。
英文原文摘要 1286 字符
The emergence of Large Language Models (LLMs) has driven rapid progress in multi-modal learning, particularly in the development of Large Vision-Language Models (LVLMs). However, existing LVLM training paradigms place excessive reliance on the LLM component, giving rise to two critical robustness challenges: language bias and language sensitivity. To address both issues simultaneously, we propose a novel Self-Critical Inference (SCI) framework that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations. This process further introduces a new strategy for improving robustness by scaling the number of counterfactual rounds. Moreover, we also observe that failure cases of LVLMs differ significantly across models, indicating that fixed robustness benchmarks may not be able to capture the true reliability of LVLMs. To this end, we propose the Dynamic Robustness Benchmark (DRBench), a model-specific evaluation framework targeting both language bias and sensitivity issues. Extensive experiments show that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.