看见、说出、理清:面向 LVLM 视觉接地多模态推理的迭代式免训练框架
See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
提出一种免训练、即插即用的测试时逐步视觉证据监督机制:构建文本化视觉证据池,并用视觉决策模块按推理上下文动态补充证据,可直接套用到任意 LVLM 骨干上抑制多模态思维链中的视觉幻觉传播,无需 RL 微调。
原文依据 we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期的大型视觉语言模型(LVLM)通过生成长思维链(CoT)回答展现出令人印象深刻的推理能力。然而,多模态语境下的思维链推理极易受到视觉幻觉传播的影响:一旦某个中间推理步骤与视觉证据不一致,后续步骤即便逻辑上成立,仍可能导致最终答案错误。现有解决方案试图通过强化学习(RL)训练模型「用图像思考」来缓解该问题。这类方法虽然有效,但代价高昂、依赖特定模型,且难以跨架构泛化。与之不同,我们提出一种轻量级方法,绕过 RL 训练,为视觉接地的多模态推理提供迭代式、免训练、即插即用的框架。我们的核心思想是在测试时用视觉证据监督每一个推理步骤,确保每个解码出的词元都能由相应的视觉线索所支撑。具体而言,我们构建一个文本化的视觉证据池来引导模型的推理生成。当已有证据不足时,一个视觉决策模块会根据正在进行的推理上下文,从图像中动态提取额外的相关证据,扩展证据池,直至模型获得足够的视觉确定性以终止推理并给出最终答案。在多个 LVLM 骨干与基准上的大量实验证明了本方法的有效性。我们的方法在 TreeBench 上取得 16.5%–29.5% 的提升,在 RH-Bench 上取得 13.7% 的 RH-AUC 增益,在无需额外训练的情况下显著降低幻觉率并提升推理准确率。
英文原文摘要 1698 字符
Recent large vision-language models (LVLMs) have demonstrated impressive reasoning ability by generating long chain-of-thought (CoT) responses. However, CoT reasoning in multimodal contexts is highly vulnerable to visual hallucination propagation: once an intermediate reasoning step becomes inconsistent with the visual evidence, subsequent steps-even if logically valid-can still lead to incorrect final answers. Existing solutions attempt to mitigate this issue by training models to "think with images" via reinforcement learning (RL). While effective, these methods are costly, model-specific, and difficult to generalize across architectures. Differently, we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning. Our key idea is to supervise each reasoning step at test time with visual evidence, ensuring that every decoded token is justified by corresponding visual cues. Concretely, we construct a textual visual-evidence pool that guides the model's reasoning generation. When existing evidence is insufficient, a visual decider module dynamically extracts additional relevant evidence from the image based on the ongoing reasoning context, expanding the pool until the model achieves sufficient visual certainty to terminate reasoning and produce the final answer. Extensive experiments on multiple LVLM backbones and benchmarks demonstrate the effectiveness of our approach. Our method achieves 16.5%-29.5% improvements on TreeBench and 13.7% RH-AUC gains on RH-Bench, substantially reducing hallucination rates while improving reasoning accuracy without additional training.