MoE-VLM 中软模态引导的专家特化
Soft Modality-Guided Expert Specialization in MoE-VLMs
对做多模态大模型的人来说，这篇给出了一个可复用的路由改造思路：用随层变化的软模态分数（注意力或高斯统计得到）驱动专家分配，并加互信息正则化让专家按模态分工，同时按专家并行部署做分箱以降低通信；可直接迁移到任意 MoE-VLM 的专家路由设计中。
原文依据 We propose Soft Modality-guided Expert Specialization (SMoES), which consists of dynamic soft modality scores that capture layer-dependent fusion patterns
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论混合专家（MoE）已成为大型视觉语言模型（VLM）的主流骨干网络，但模态特定信号应如何引导专家路由仍缺乏探索。现有路由策略要么是手工设计的，要么与模态无关，依赖理想化先验，忽略了 MoE-VLM 中随层变化的模态融合模式，对专家特化几乎没有指导作用。我们提出软模态引导的专家特化（SMoES），它包含捕捉随层融合模式的动态软模态分数、与专家并行部署对齐的专家分箱机制，以及鼓励一致的模态特化的箱间互信息正则化。我们的方法利用基于注意力或高斯统计的模态分数来优化互信息正则化。在四个基于 MoE 的 VLM 和 16 个基准上的实验表明，该方法在有效性和效率上均有提升：多模态任务和纯语言任务平均分别提升 0.9% 和 4.2%，专家并行通信开销降低 56.1%，真实部署下吞吐量提升 12.3%。这些结果验证了将路由与模态感知的专家特化对齐能够释放 MoE-VLM 的容量与效率。
英文原文摘要 1278 字符
Mixture-of-Experts (MoE) has become a prevalent backbone for large vision-language models (VLMs), yet how modality-specific signals should guide expert routing remains under-explored. Existing routing strategies are either hand-crafted or modality-agnostic, relying on idealized priors that ignore the layer-dependent modality fusion patterns in MoE-VLMs and provide little guidance for expert specialization. We propose Soft Modality-guided Expert Specialization (SMoES), which consists of dynamic soft modality scores that capture layer-dependent fusion patterns, an expert binning mechanism aligned with expert-parallel deployment, and an inter-bin mutual information regularization that encourages coherent modality specialization. Our method leverages attention-based or Gaussian-statistics modality scores to optimize mutual information regularization. Experiments across four MoE-based VLMs and 16 benchmarks demonstrate improvement on both effectiveness and efficiency: 0.9% and 4.2% average gain on multimodal and language-only tasks, 56.1% reduction in EP communication overhead, and 12.3% throughput improvement under realistic deployment. These results validate that aligning routing with modality-aware expert specialization unlocks MoE-VLM capacity and efficiency.