MoE-GRPO：在视觉语言模型中通过强化学习优化混合专家
MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models
把 MoE 视觉语言模型的专家路由从确定性 top-K 改成用 GRPO 学习的随机策略，并加入模态感知的路由引导；做多模态大模型推理或训练的人可以直接复用这套路由优化思路来缓解专家过拟合、提升专家利用率。
原文依据 we formulate expert selection as a sequential decision-making problem and optimize it using Group Relative Policy Optimization (GRPO)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论混合专家（MoE）已成为降低 Transformer 架构计算开销的有效途径：它为每个词元稀疏地激活一部分参数，同时保持较高的模型容量。这一范式近来被扩展到视觉语言模型（VLM），使得在降低计算成本的同时实现可扩展的多模态理解。然而，广泛采用的确定性 top-K 路由机制可能忽略更优的专家组合，并导致专家过拟合。为解决这一局限并提升专家选择的多样性，我们提出 MoE-GRPO，一个基于强化学习（RL）的框架，用于优化基于 MoE 的 VLM 中的专家路由。具体而言，我们将专家选择建模为序列决策问题，并使用组相对策略优化（GRPO）对其进行优化，使模型能够通过探索与基于奖励的反馈学习自适应的专家路由策略。此外，我们引入模态感知的路由器引导，通过抑制路由器去探索对给定模态而言激活频率较低的专家，提升训练的稳定性与效率。在多模态图像与视频基准上的大量实验表明，MoE-GRPO 通过促进更多样的专家选择，持续优于标准 top-K 路由及其变体，从而缓解专家过拟合并实现任务级的专家特化。
英文原文摘要 1459 字符
Mixture-of-Experts (MoE) has emerged as an effective approach to reduce the computational overhead of Transformer architectures by sparsely activating a subset of parameters for each token while preserving high model capacity. This paradigm has recently been extended to Vision-Language Models (VLMs), enabling scalable multi-modal understanding with reduced computational cost. However, the widely adopted deterministic top-K routing mechanism may overlook more optimal expert combinations and lead to expert overfitting. To address this limitation and improve the diversity of expert selection, we propose MoE-GRPO, a reinforcement learning (RL)-based framework for optimizing expert routing in MoE-based VLMs. Specifically, we formulate expert selection as a sequential decision-making problem and optimize it using Group Relative Policy Optimization (GRPO), allowing the model to learn adaptive expert routing policies through exploration and reward-based feedback. Furthermore, we introduce a modality-aware router guidance that enhances training stability and efficiency by discouraging the router from exploring experts that are infrequently activated for a given modality. Extensive experiments on multi-modal image and video benchmarks show that MoE-GRPO consistently outperforms standard top-K routing and its variants by promoting more diverse expert selection, thereby mitigating expert overfitting and enabling a task-level expert specialization.