以感知为中心的过程奖励模型改进视觉语言模型
Improving Vision-language Models with Perception-centric Process Reward Models
提供了一种可复用的过程奖励模型思路：把 VLM 回答拆成与图像相关的断言，逐条与视觉证据比对以定位感知幻觉片段，并将该信号转为 token 级优势用于 RL 训练或推理时的截断重生成，可直接用于缓解视觉幻觉与提升多模态推理。
原文依据 extract image-related claims from the response and compare them one by one with the visual evidence in the image
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期，带可验证奖励的强化学习（RLVR）显著提升了视觉语言模型（VLM）的复杂推理能力。然而，其结果级监督过于粗糙，难以诊断并纠正推理链内部的错误。为此，我们提出 Perceval，一种过程奖励模型（PRM），它能够实现 token 级错误定位：从回答中抽取与图像相关的断言，并将其与图像中的视觉证据逐一比对，最终返回包含感知错误的断言。Perceval 使用感知密集型监督训练数据进行训练。随后，我们将 Perceval 集成到强化学习训练过程中以训练策略模型。具体而言，与采用序列级优势的传统 GRPO 相比，我们通过针对 Perceval 所识别的幻觉片段施加惩罚来实现 token 级优势，从而获得细粒度的监督信号。除增强训练过程外，Perceval 还能在推理阶段辅助 VLM。借助 Perceval，我们可以截断模型回答中的错误部分，然后让模型直接重新生成回答，或引导模型反思其先前的输出。该过程可重复多次以实现测试时扩展。实验表明，在多个经强化学习训练的推理 VLM 上，该方法在各领域基准上均取得显著提升。在测试时扩展方面，相较于多数投票等其他策略，它也表现出一致的性能增益。我们的代码与数据将公开发布。
英文原文摘要 1626 字符
Recent advancements in reinforcement learning with verifiable rewards (RLVR) have significantly improved the complex reasoning ability of vision-language models (VLMs). However, its outcome-level supervision is too coarse to diagnose and correct errors within the reasoning chain. To this end, we propose Perceval, a process reward model (PRM) that enables token-level error grounding, which can extract image-related claims from the response and compare them one by one with the visual evidence in the image, ultimately returning claims that contain perceptual errors. Perceval is trained with perception-intensive supervised training data. We then integrate Perceval into the RL training process to train the policy models. Specifically, compared to traditional GRPO, which applies sequence-level advantages, we apply token-level advantages by targeting penalties on hallucinated spans identified by Perceval, thus enabling fine-grained supervision signals. In addition to augmenting the training process, Perceval can also assist VLMs during the inference stage. Using Perceval, we can truncate the erroneous portions of the model's response, and then either have the model regenerate the response directly or induce the model to reflect on its previous output. This process can be repeated multiple times to achieve test-time scaling. Experiments show significant improvements on benchmarks from various domains across multiple reasoning VLMs trained with RL. For test-time scaling, it also demonstrates consistent performance gains over other strategies, such as major voting. Our code and data will be publicly released.