通过免训练奖励归因解锁词元奖励
Unlocking Token Rewards via Training-Free Reward Attribution
对做多模态大模型推理（如 MathVista 上的视觉数学推理）的研究者，这篇给出了一个免训练、单次前向+反向即可从现有 PRM 中提取词元级奖励的方法，可直接替换 RL 微调中的结果奖励做更细粒度的信用分配，无需再训练奖励模型。
原文依据 we use a first-order Taylor approximation, which simplifies the influence calculation to the inner product of the difference between the token embedding and the null token embedding
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文提出一种极其高效、免训练的方法，可直接从已有的深度奖励模型中提取词元级奖励信号。其核心思想是通过估计每个词元的影响，将整体过程奖励归因到各个词元上。该影响被定义为：当某个词元被替换为语义空词元时，最终宏观奖励（例如过程奖励）所发生的变化。直接计算这一影响在计算上不可行，因为对于长度为 N 的词元序列，需要对 PRM 进行 N 次前向传播。我们通过提出一种高效的基于梯度的估计器来突破这一瓶颈。具体而言，我们采用一阶泰勒近似，将影响计算简化为词元嵌入与空词元嵌入之差同奖励对词元嵌入的梯度的内积。这仅需一次前向和一次反向传播。由此得到的词元级奖励使标准强化学习算法能够在无需额外训练奖励模型的情况下执行精确的信用分配。在具有挑战性的推理基准上的实验表明，我们的方法显著提升了策略优化效率，并增强了 LLM 推理能力的泛化性。对于 Qwen2.5-VL-7B-Instruct，我们的 P2T 在 MathVista 上比结果奖励高出 +4.9%；对于 Qwen2.5-Math-7B，在 AIME24 上高出 +11.5%。代码见：https://github.com/JIA-Lab-research/P2T。
英文原文摘要 1509 字符
In this paper, we propose an extremely efficient, training-free method to extract token-level reward signals directly from an existing deep reward model. Our core idea is to attribute the overall process reward to individual tokens by estimating each token's influence. This influence is defined as the change in the final macroscopic reward (e.g., the process reward) when a token is replaced with a semantically null token. Naively calculating this influence is computationally infeasible, requiring N forward passes through the PRM for an N-token sequence. We overcome this bottleneck by proposing a highly efficient gradient-based estimator. Specifically, we use a first-order Taylor approximation, which simplifies the influence calculation to the inner product of the difference between the token embedding and the null token embedding, and the gradient of the reward with respect to the token embedding. This requires only a single forward and backward pass. The resulting token-level rewards enable standard RL algorithms to perform precise credit assignment without requiring additional reward model training. Experiments on challenging reasoning benchmarks demonstrate that our method substantially improves policy optimization efficiency and enhances the generalization of LLM reasoning capabilities. Our P2T outperforms the outcome reward by +4.9% on MathVista for Qwen2.5-VL-7B-Instruct, and +11.5% on AIME24 for Qwen2.5-Math-7B. The code is available at: https://github.com/JIA-Lab-research/P2T.