MSRL：通过多阶段强化学习扩展生成式多模态奖励建模
MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning
对做视觉生成与理解评测的人，这篇给出了一个可复用的思路：用大规模纯文本偏好数据先训练出可泛化的奖励推理能力，再经 caption 阶段和全多模态 RL 阶段迁移到视觉奖励模型，从而在缺少多模态偏好标注时也能扩展 MRM 训练；其中的跨模态蒸馏环节可直接用于提升视觉奖励模型的偏好泛化。
原文依据 we propose a Multi-Stage Reinforcement Learning (MSRL) approach, which can achieve scalable reinforcement learning for MRMs with limited multimodal data
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期多模态奖励建模的进展在很大程度上源于从判别式方法向生成式方法的范式转变。在此进展基础上，近期研究进一步采用带可验证奖励的强化学习（RLVR）来增强多模态奖励模型（MRM）。尽管取得了成功，基于 RLVR 的训练通常依赖带标注的多模态偏好数据，而这类数据的获取成本高、劳动密集，使得 MRM 的训练难以扩展。为克服这一局限，我们提出一种多阶段强化学习（MSRL）方法，能够在多模态数据有限的情况下为 MRM 实现可扩展的强化学习。MSRL 重新定义了传统的基于 RLVR 的训练范式：它首先从大规模文本偏好数据中学习可泛化的奖励推理能力，然后通过基于描述（caption）的强化学习阶段和全多模态强化学习阶段，将该能力逐步迁移到多模态任务上。此外，我们引入一种跨模态知识蒸馏方法，以提升 MSRL 内部的偏好泛化能力。大量实验表明，MSRL 有效扩展了生成式 MRM 的基于 RLVR 的训练，并显著提升了其在视觉理解与视觉生成任务上的性能（例如在 VLReward Bench 上从 68.5% 提升至 74.8%，在 GenAI-Bench 上从 69.2% 提升至 75.4%），且无需额外的多模态偏好标注。
英文原文摘要 1455 字符
Recent advances in multimodal reward modeling have been largely driven by a paradigm shift from discriminative to generative approaches. Building on this progress, recent studies have further employed reinforcement learning with verifiable rewards (RLVR) to enhance multimodal reward models (MRMs). Despite their success, RLVR-based training typically depends on labeled multimodal preference data, which are costly and labor-intensive to obtain, making it difficult to scale the training of MRMs. To overcome this limitation, we propose a Multi-Stage Reinforcement Learning (MSRL) approach, which can achieve scalable reinforcement learning for MRMs with limited multimodal data. MSRL redefines the conventional RLVR-based training paradigm by first learning a generalizable reward reasoning capability from large-scale textual preference data and then progressively transferring this capability to multimodal tasks through caption-based and fully multimodal reinforcement learning stages. Furthermore, we introduce a cross-modal knowledge distillation approach to improve preference generalization within MSRL. Extensive experiments demonstrate that MSRL effectively scales the RLVR-based training of generative MRMs and substantially improves their performance across both visual understanding and visual generation tasks (e.g., 68.5%-74.8% on VLReward Bench, 69.2%-75.4% on GenAI-Bench), without requiring additional multimodal preference annotations.