视觉语言模型 · CVPR2026 ·2026.06.03

感知证据锚定的强化学习用于多模态推理

Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning

Chi Zhang、Haibo Qiu、Qiming Zhang、Yufei Xu、Zhixiong Zeng、Siqi Yang 等 9 人

与你有关的部分

将视觉感知验证作为强化学习奖励的锚点,通过感知检查表生成辅助奖励来约束推理过程,可复用于任何需要提升视觉证据忠实度的VLM推理任务。

原文依据 PEARL first derive a perception checklist---a set of perception-oriented sub-questions with verifiable answers that probe the model's understanding of key visual evidence.
可迁移到 视觉问答多模态推理视觉幻觉缓解

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

基于可验证奖励的强化学习(RLVR)显著提升了大语言模型(LLM)的推理能力,目前正被应用于视觉语言模型(VLM)。然而,面向VLM的普通RLVR仅验证最终文本输出,严重忽视了视觉感知这一基础步骤。这种疏忽会导致视觉幻觉和奖励黑客行为,因为建立在错误感知之上的推理本质上不可靠。为解决此问题,我们提出PEARL(感知证据锚定的强化学习),一种双分支、感知-推理协同的方法,通过将多模态推理显式锚定到已验证的视觉证据来增强其能力。对于每个面向推理的问答实例,PEARL首先推导出一个感知检查表——一组具有可验证答案的感知导向子问题,用于探查模型对关键视觉证据的理解。在训练过程中,基于该检查表的辅助推演会产生一个感知奖励,该奖励既直接强化模型的感知能力,又作为推理的忠实性门控。如果模型通过感知检查,其策略更新将偏向于证据锚定的推理;否则,过程将中止,以防止基于错误前提的推理。PEARL可以无缝集成到GRPO和DAPO等主流强化学习方法中。大量实验表明,PEARL在多模态推理基准上取得了显著提升,例如在MathVerse上相比基线提升+9.7%,相比GRPO提升+6.6%。

英文原文摘要 1534 字符

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) and is now being applied to Vision-Language Models (VLMs). However, vanilla RLVR for VLMs verifies only the final textual output, critically neglecting the foundational step of visual perception. This oversight leads to visual hallucinations and reward hacking, as reasoning built upon flawed perception is inherently unreliable. To address this, we propose PEARL (Perceptual-Evidence Anchored Reinforced Learning), a dual-branch, perception-reasoning synergistic that strengthens multimodal reasoning by explicitly anchoring it to verified visual evidence. For each reasoning-oriented QA instance, PEARL first derive a perception checklist---a set of perception-oriented sub-questions with verifiable answers that probe the model's understanding of key visual evidence. During training, auxiliary rollouts on this checklist yield a perceptual reward that both directly reinforces the model's perception ability and acts as a fidelity gate for reasoning. If the model passes the perception check, its policy update is biased towards evidence-anchored reasoning. Otherwise, the process is halted to prevent reasoning from flawed premises. PEARL can be seamlessly integrated with popular RL methods like GRPO and DAPO. Comprehensive experiments show PEARL achieves substantial gains on multimodal reasoning benchmarks, e.g., a +9.7% improvement over the baseline and +6.6% over GRPO on MathVerse.

标签

会议 CVPR2026
页码 41111-41120
Bibkey
来源 CVF Open Access