殊途同归：激励视觉语言模型中的发散性思维
All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models
对做 VLM 推理增强的人，这篇给出了 GRPO 多样性坍塌的诊断（模型过早收敛到少数推理策略），并提出 MUPO 用多组策略优化维持解空间多样性，可直接替换 GRPO 的训练目标用于多模态推理微调。
原文依据 GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期研究表明，强化学习（RL），尤其是组相对策略优化（GRPO），能够内在地激发并增强视觉语言模型（VLM）的推理能力。然而，尽管前景可观，驱动 RL 模型有效性的内在机制及其局限性仍缺乏深入探索。本文中，我们强调了 RL 模型与基座模型之间一个根本性的行为差异：前者进行更深但更窄的推理，而基座模型尽管在单条路径上不够精细，却展现出更广、更多样的思维模式。通过进一步分析训练动态，我们表明 GRPO 容易出现多样性坍塌，使模型过早收敛到有限的推理策略子集，同时丢弃大多数潜在备选方案，从而导致局部最优与较差的可扩展性。为解决这一问题，我们提出多组策略优化（MUPO），一种简单而有效的方法，旨在激励跨多个解的发散性思维，并在已有基准上验证了其有效性。
英文原文摘要 1143 字符
Recent studies have demonstrated that Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO), can intrinsically elicit and enhance the reasoning capabilities of Vision-Language Models (VLMs). However, despite the promise, the underlying mechanisms that drive the effectiveness of RL models as well as their limitations remain underexplored. In this paper, we highlight a fundamental behavioral distinction between RL and base models, where the former engages in deeper yet narrow reasoning, while base models, despite less refined along individual path, exhibit broader and more diverse thinking patterns. Through further analysis of training dynamics, we show that GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies while discarding the majority of potential alternatives, leading to local optima and poor scalability. To address this, we propose Multi-Group Policy Optimization (MUPO), a simple yet effective approach designed to incentivize divergent thinking across multiple solutions, and demonstrate its effectiveness on established benchmarks.