通过聚类感知上采样增强混合专家模型的特化能力
Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling
对需要在视觉-语言预训练中把稠密 CLIP 骨干升级为 MoE 的研究者，本文给出一种可复用的初始化方案：用输入激活的语义聚类 + 截断 SVD 初始化各专家、用聚类质心初始化路由器，并配合专家集成自蒸馏稳定路由，可直接替换稀疏上采样以提升零样本/少样本迁移。
原文依据 Each expert is then initialized using the subspace representations of its corresponding cluster via truncated SVD, while setting the router's initial weights to the cluster centroids.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论稀疏上采样提供了一种从预训练稠密权重初始化混合专家（MoE）模型、而非从头训练的高效方式。然而，由于所有专家都从相同权重出发且路由器随机初始化，模型会受困于专家对称性与早期特化不足的问题。我们提出聚类感知上采样，一种将语义结构引入 MoE 初始化的策略。该方法首先将稠密模型的输入激活划分为语义簇；随后借助截断 SVD，用每个专家对应簇的子空间表示来初始化该专家，同时将路由器的初始权重设为各簇质心。这种聚类感知初始化打破了专家对称性，并促使专家早期形成与数据分布一致的特化。此外，我们引入专家集成自蒸馏损失，通过集成教师提供可靠的路由指导来稳定训练。在 CLIP ViT-B/32 与 ViT-B/16 上的评测表明，聚类感知上采样在零样本与少样本基准上均持续优于现有方法。该方法还产生了更多样、更解耦的专家表示，降低了专家间相似度，并带来更自信的路由行为。
英文原文摘要 1320 字符
Sparse Upcycling provides an efficient way to initialize a Mixture-of-Experts (MoE) model from pretrained dense weights instead of training from scratch. However, since all experts start from identical weights and the router is randomly initialized, the model suffers from expert symmetry and limited early specialization. We propose Cluster- aware Upcycling, a strategy that incorporates semantic structure into MoE initialization. Our method first partitions the dense model's input activations into semantic clusters. Each expert is then initialized using the subspace representations of its corresponding cluster via truncated SVD, while setting the router's initial weights to the cluster centroids. This cluster-aware initialization breaks expert symmetry and encourages early specialization aligned with the data distribution. Furthermore, we introduce an expert-ensemble self-distillation loss that stabilizes training by providing reliable routing guidance using an ensemble teacher. When evaluated on CLIP ViT-B/32 and ViT-B/16, Cluster-aware Upcycling consistently outperforms existing methods across both zero-shot and few-shot benchmarks. The proposed method also produces more diverse and dis- entangled expert representations, reduces inter-expert similarity, and leads to more confident routing behavior.