论词元的困境：面向大型视觉语言模型持续学习的漂移感知词元分配动态混合专家
On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models
对做 LVLM 持续指令微调的人，这篇给出了 token 级诊断视角：用路由分数分布区分歧义/旧任务词元，并对它们施加定向正则，避免被新专家吸走；这套「漂移感知词元分配」可直接插到任何增量扩专家的 MoE-LVLM 上。
原文依据 old-task tokens become mistakenly attracted to newly added experts, degrading performance on prior tasks
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态持续指令微调旨在通过从新数据中学习来持续增强大型视觉语言模型（LVLM），同时不遗忘先前已获得的知识。混合专家（MoE）架构通过增量添加新专家并扩展路由器、同时保持已有专家冻结，天然地支持这一目标。然而，尽管专家之间相互隔离，基于 MoE 的持续学习器仍会因路由漂移而遭受遗忘：旧任务的词元会被错误地吸引到新增专家上，从而损害先前任务的性能。我们在词元层面分析了这一失效模式，并揭示了「词元的困境」：新任务数据中的歧义词元和旧词元带来的学习收益极小，但由于训练过程中其路由分配存在歧义，一旦被路由到新专家便会引发遗忘。受此启发，我们提出 LLaVA-DyMoE，一个以漂移感知词元分配进行增量式 MoE 扩展的动态 MoE 框架。我们通过词元的路由分数分布来刻画其类型，并施加有针对性的正则化。具体而言，词元级分配引导将歧义词元和旧词元推离新专家，以保持既有的路由模式并缓解路由漂移；互补的路由分数正则化则强制专家组之间相互分离，并促进新专家的专门化。大量实验表明，我们的 LLaVA-DyMoE 有效缓解了路由漂移引发的遗忘，相比基线在平均最终准确率上取得超过 7% 的提升，遗忘率降低 12%。项目主页为 zhaoc5.github.io/DyMoE。
英文原文摘要 1616 字符
Multimodal Continual Instruction Tuning aims to continually enhance Large Vision Language Models (LVLMs) by learning from new data without forgetting previously acquired knowledge. Mixture of Experts (MoE) architectures naturally facilitate this by incrementally adding new experts and expanding routers while keeping the existing ones frozen. However, despite expert isolation, MoE-based continual learners still suffer from forgetting due to routing-drift: old-task tokens become mistakenly attracted to newly added experts, degrading performance on prior tasks. We analyze the failure mode at the token level and reveal the token's dilemma: ambiguous and old tokens in new-task data offer minimal learning benefit yet induce forgetting when routed to new experts, due to their ambiguous routing assignment during training. Motivated by this, we propose LLaVA-DyMoE, a dynamic MoE framework that incrementally expands the MoE with drift-aware token assignment. We characterize token types via their routing score distributions and apply targeted regularization. Specifically, a token-level assignment guidance steers ambiguous and old tokens away from new experts to preserve established routing patterns and alleviate routing-drift, while complementary routing score regularizations enforce expert-group separation and promote new-expert specialization. Extensive experiments demonstrate that our LLaVA-DyMoE effectively mitigates routing-drift-induced forgetting, achieving over a 7% gain in mean final accuracy and a 12% reduction in forgetting compared to baselines. The project page is zhaoc5.github.io/DyMoE.