从探索到利用：面向噪声容忍 MLLM 训练的两阶段熵 RLVR 方法
From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
对做 MLLM 后训练的人，这套「先熵最大化探索、后熵最小化利用」的 token 级熵调度可直接插到 GRPO 训练循环里，用来在标注噪声大的视觉问答/多模态推理数据上稳住组内奖励方差、避免过早收敛到错标。
原文依据 we propose a novel two-stage, token-level entropy optimization method for RLVR
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论面向多模态大语言模型（MLLM）的带可验证奖励的强化学习（RLVR）高度依赖高质量标注数据，而真实场景中这类数据往往稀缺且易含大量标注噪声。现有的无监督 RLVR 方法（包括纯熵最小化）会过拟合到错误标签，并削弱组相对策略优化（GRPO）所依赖的关键奖励排序信号。为应对这些挑战并提升噪声容忍能力，我们提出一种新的两阶段、token 级熵优化 RLVR 方法。该方法在训练过程中动态引导模型从探索走向利用。在初始的探索阶段，token 级熵最大化促使输出生成多样且具随机性，充当强正则项，防止模型过早收敛到噪声标签，并保证组内足够的差异性，从而使 GRPO 中的奖励梯度估计更可靠。随着训练推进，方法转入利用阶段，token 级熵最小化促使模型输出置信且确定的结果，从而巩固已获得的知识并提升预测精度。在实验中，我们在 Qwen2-VL-2B、Qwen2-VL-7B 与 Qwen2.5-VL-3B 三个 MLLM 骨干上，覆盖多种噪声设定与多项任务，该分阶段策略通过统一并增强外部、内部与基于熵的方法，全面优于先前方法，在各设定下均取得稳健且更优的性能。
英文原文摘要 1563 字符
Reinforcement Learning with Verifiable Rewards (RLVR) for Multimodal Large Language Models (MLLMs) is highly dependent on high-quality labeled data, which is often scarce and prone to substantial annotation noise in real-world scenarios. Existing unsupervised RLVR methods, including pure entropy minimization, can overfit to incorrect labels and limit the crucial reward ranking signal for Group-Relative Policy Optimization (GRPO). To address these challenges and enhance noise tolerance, we propose a novel two-stage, token-level entropy optimization method for RLVR. This approach dynamically guides the model from exploration to exploitation during training. In the initial exploration phase, token-level entropy maximization promotes diverse and stochastic output generation, serving as a strong regularizer that prevents premature convergence to noisy labels and ensures sufficient intra-group variation--enabling more reliable reward gradient estimation in GRPO. As training progresses, the method transitions into the exploitation phase, where token-level entropy minimization encourages the model to produce confident and deterministic outputs, thereby consolidating acquired knowledge and refining prediction accuracy. Empirically, across three MLLM backbones-Qwen2-VL-2B, Qwen2-VL-7B, and Qwen2.5-VL-3B--spanning diverse noise settings and multiple tasks, our phased strategy consistently outperforms prior approaches by unifying and enhancing external, internal, and entropy-based methods, delivering robust and superior performance across the board.