PDCR：面向视觉语言推理的感知分解置信度奖励
PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning
对做 VLM 推理训练的人，这篇给出了一个可直接复用的奖励塑形思路：用模型内部的 Visual Dependence Score 把推理链中的视觉感知步与文本推理步分开，再在各自簇内归一化置信度增益，避免稀疏视觉步的信号被密集文本步淹没；该分解与归一化流程可迁移到任何带思维链的多模态强化学习训练中。
原文依据 This global normalization creates mixture-induced signal degradation, where the training signal for visual steps is statistically distorted by the predominant textual steps.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论可验证奖励强化学习（RLVR）传统上依赖稀疏的、基于结果的信号。近期工作表明，提供一种细粒度的、模型内在的信号——即奖励模型对正确答案的置信度增长——能够在不依赖昂贵外部模型的情况下提供步骤级引导，从而有效改进语言推理训练。虽然该做法在单模态文本上有效，但我们发现，将这种全局奖励朴素地应用于视觉语言（V-L）推理并非最优策略，因为该任务是稀疏视觉感知与密集文本推理的异质混合。这种全局归一化会造成混合导致的信号退化：视觉步骤的训练信号会被占主导的文本步骤在统计上扭曲。我们提出感知分解置信度奖励（PDCR），该框架通过使奖励结构与任务的异质本性相对齐来解决这一问题。PDCR 首先执行无监督技能分解，引入模型内部的视觉依赖度分数来量化视觉依赖程度，并应用聚类算法将感知步骤与推理步骤分离。在此基础上，PDCR 通过在每个技能簇内归一化置信度增益来计算分解后的优势。这种簇内归一化为感知与推理两者都提供了稳定且尺度正确的信号。我们在关键 V-L 推理基准上证明，PDCR 优于朴素的全局奖励形式以及稀疏奖励基线。
英文原文摘要 1501 字符
Reinforcement Learning with Verifiable Rewards (RLVR) traditionally relies on a sparse, outcome-based signal. Recent work shows that providing a fine-grained, model-intrinsic signal--rewarding the confidence growth in the ground-truth answer--effectively improves language reasoning training by providing step-level guidance without costly external models. While effective for unimodal text, we find that naively applying this global reward to vision-language (V-L) reasoning is a suboptimal strategy, as the task is a heterogeneous mix of sparse visual perception and dense textual reasoning. This global normalization creates mixture-induced signal degradation, where the training signal for visual steps is statistically distorted by the predominant textual steps. We propose Perception-Decomposed Confidence Reward (PDCR), a framework that solves this by aligning the reward structure with the task's heterogeneous nature. PDCR first performs an unsupervised skill decomposition, introducing a model-internal Visual Dependence Score to quantify visual reliance and applying a clustering algorithm to separate perception and reasoning steps. Based on this, PDCR computes a decomposed advantage by normalizing confidence gains within each skill cluster. This intra-cluster normalization provides a stable, correctly-scaled signal for both perception and reasoning. We demonstrate that PDCR outperforms the naive, global-reward formulation and sparse-reward baselines on key V-L reasoning benchmarks.