ERMoE：用于稳定路由与可解释专家特化的特征基重参数化混合专家模型
ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization
用专家表示空间的正交特征基余弦相似度替代可学习门控 logits 做路由，可作为一种免辅助负载均衡损失的 MoE 路由方案，直接迁移到视觉 Transformer 的稀疏扩容与多模态检索骨干中。
原文依据 routes with an Eigenbasis Score (cosine similarity between token features and an expert basis) instead of learned gating logits
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论混合专家（MoE）模型通过稀疏专家激活扩展容量，但路由 logits 可能与专家结构失配（路由不稳定、专家利用不足），且负载不均衡会造成长尾拖累。辅助负载均衡损失可减小差异，但往往削弱专家特化并降低下游精度。我们提出 ERMoE，一种稀疏 MoE Transformer，它在学习到的正交特征基中重参数化每个专家，并用特征基得分（词元特征与专家基之间的余弦相似度）进行路由，而非使用可学习的门控 logits。通过将分配与每个专家的表示空间绑定，ERMoE 稳定了专家利用率、提升了可解释性，并去除了显式均衡损失及其梯度干扰。ERMoE 在 ImageNet 与图像-文本检索（COCO、Flickr30K）上取得最优结果，且专家负载更均衡。一个 3D MRI 变体（ERMoE-ba）将脑龄预测提升超过 7%，并给出在解剖学上可解释的专家特化。
英文原文摘要 1001 字符
Mixture-of-Experts (MoE) models expand capacity via sparse expert activation, but routing logits can misalign with expert structure (unstable routing, underutilization) and load imbalance can create stragglers. Auxiliary load-balancing losses reduce disparity but often weaken specialization and downstream accuracy. We propose ERMoE, a sparse MoE transformer that reparameterizes each expert in a learned orthonormal eigenbasis and routes with an Eigenbasis Score (cosine similarity between token features and an expert basis) instead of learned gating logits. By tying assignments to each expert's representation space, ERMoE stabilizes utilization, improves interpretability, and removes explicit balancing losses and their gradient interference. ERMoE reaches state-of-the-art results on ImageNet and image-text retrieval (COCO, Flickr30K) with flatter expert loads. A 3D MRI variant (ERMoE-ba) improves brain age prediction by over 7% and yields anatomically interpretable expert specializations.