稳定高效的单次采样强化学习用于多模态推理
Stable and Efficient Single-Rollout RL for Multimodal Reasoning
针对多模态大模型强化学习微调中单次采样不稳定的问题，提出基于熵的优势塑造机制，可复用于视觉语言模型的RLVR训练，提升样本效率并防止训练崩溃。
原文依据 MSSR achieves this via an entropy-based advantage-shaping mechanism that adaptively regularizes advantage magnitudes, preventing collapse and maintaining training stability.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于可验证奖励的强化学习（RLVR）已成为提升多模态大语言模型（MLLMs）推理能力的关键范式。然而，主流的基于组的算法如GRPO需要对每个提示进行多次采样。尽管最近在纯文本设置中探索了更高效的单次采样变体，但我们发现它们在多模态语境下会遭遇严重的不稳定性，常常导致训练崩溃。为了解决这种样本效率与稳定性之间的权衡，我们引入了MSSR（多模态稳定单次采样），一个无组的RLVR框架，同时实现稳定优化和有效的多模态推理性能。MSSR通过基于熵的优势塑造机制实现这一点，该机制自适应地正则化优势幅度，防止崩溃并保持训练稳定性。虽然此类机制已在基于组的RLVR中使用，但我们表明在多模态单次采样设置中，它们不仅有益而且是稳定性的必要条件。在分布内评估中，MSSR展现出优越的采样样本效率，以一半的训练步数达到相似的验证准确率。当训练相同步数时，MSSR的性能超越基于组的基线，并在五个多样化的推理密集型基准上显示出持续的泛化改进。这些结果共同表明，MSSR为复杂的多模态推理任务实现了稳定、样本高效且有效的RLVR。
英文原文摘要 1535 字符
Reinforcement Learning with Verifiable Rewards (RLVR) has become a key paradigm to improve the reasoning capabilities of Multimodal Large Language Models (MLLMs). However, prevalent group-based algorithms such as GRPO require multi-rollout sampling for each prompt. While more efficient single-rollout variants have recently been explored in text-only settings, we find that they suffer from severe instability in multimodal contexts, often leading to training collapse. To address this sample efficiency-stability trade-off, we introduce MSSR (Multimodal Stabilized Single-Rollout), a group-free RLVR framework that achieves both stable optimization and effective multimodal reasoning performance. MSSR achieves this via an entropy-based advantage-shaping mechanism that adaptively regularizes advantage magnitudes, preventing collapse and maintaining training stability. While such mechanisms have been used in group-based RLVR, we show that in the multimodal single-rollout setting they are not merely beneficial but essential for stability. In in-distribution evaluations, MSSR demonstrates superior rollout sample efficiency, achieving similar validation accuracy with half the training steps. When trained for the same number of steps, MSSR's performance surpasses the group-based baseline and shows consistent generalization improvements across five diverse reasoning-intensive benchmarks. Together, these results demonstrate that MSSR enables stable, sample-efficient, and effective RLVR for complex multimodal reasoning tasks.