教师引导路由的稀疏视觉混合专家
Teacher-Guided Routing for Sparse Vision Mixture-of-Experts
提出用预训练稠密教师模型的中间表示构建教师路由器，为学生路由器的专家选择提供伪监督，从而稳定稀疏MoE训练中的路由波动；这一思路可直接迁移到视觉MoE模型（如V-MoE）的训练中，提升高稀疏配置下的路由一致性与精度。
原文依据 TGR-MoE constructs a teacher router from the teacher's intermediate representations and uses its routing outputs as pseudo-supervision for the student router
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期深度学习的进展由日益增大的模型规模驱动，但由此产生的计算成本已成为关键瓶颈。稀疏混合专家（MoE）通过为每个输入仅激活一小部分专家，在不牺牲推理速度的前提下实现高可扩展性，提供了有效解决方案。尽管有效，稀疏MoE训练表现出特有的优化困难。由于路由器仅通过前向传播中被选中的专家接收有信息的梯度，它遭受梯度阻塞，从未被选中的路由中获得的信息很少。这种有限且高度局部的反馈使路由器难以学习合适的专家选择分数，并常导致不稳定的路由动态，例如训练过程中专家分配的波动。为解决此问题，我们提出TGR-MoE：教师引导路由的稀疏视觉混合专家，一种简单而有效的方法，利用来自预训练稠密教师模型的监督来稳定路由器学习。TGR-MoE从教师的中间表示构建教师路由器，并将其路由输出作为学生路由器的伪监督，抑制训练过程中频繁的路由波动，并在训练早期实现知识引导的专家选择。在ImageNet-1K和CIFAR-100上的大量实验表明，TGR持续提升准确率和路由一致性，同时即使在高度稀疏配置下也保持稳定训练。
英文原文摘要 1585 字符
Recent progress in deep learning has been driven by increasingly large-scale models, but the resulting computational cost has become a critical bottleneck. Sparse Mixture of Experts (MoE) offers an effective solution by activating only a small subset of experts for each input, achieving high scalability without sacrificing inference speed. Although effective, sparse MoE training exhibits characteristic optimization difficulties. Because the router receives informative gradients only through the experts selected in the forward pass, it suffers from gradient blocking and obtains little information from unselected routes. This limited, highly localized feedback makes it difficult for the router to learn appropriate expert-selection scores and often leads to unstable routing dynamics, such as fluctuating expert assignments during training. To address this issue, we propose TGR-MoE: Teacher-Guided Routing for Sparse Vision Mixture-of-Experts, a simple yet effective method that stabilizes router learning using supervision derived from a pretrained dense teacher model. TGR-MoE constructs a teacher router from the teacher's intermediate representations and uses its routing outputs as pseudo-supervision for the student router, suppressing frequent routing fluctuations during training and enabling knowledge-guided expert selection from the early stages of training. Extensive experiments on ImageNet-1K and CIFAR-100 demonstrate that TGR consistently improves both accuracy and routing consistency, while maintaining stable training even under highly sparse configurations.