面向鲁棒多模态语义分割的教师-学生框架与混合原型蒸馏
Towards Robust Multi-Modal Semantic Segmentation with Teacher-Student Framework and Hybrid Prototype Distillation
面向缺失模态的多模态语义分割，用反馈式自蒸馏让教师从非主导模态获得学生反馈，并用高层语义原型做跨模态蒸馏、用类内特征变化蒸馏保留模态特有细节，可直接迁移到任意多模态融合分割骨干上提升缺失模态鲁棒性。
原文依据 we make the student model provide feedback from the non-dominant modality to the teacher
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论由于传感器数据不完整、退化或缺失，多模态语义分割（MMSS）在实际应用中面临重大挑战。为此，我们提出 RobustSeg，一种高效的教师-学生框架，在缺失模态条件下增强模型鲁棒性，同时在完整模态场景下保持强劲性能。RobustSeg 采用基于反馈的自蒸馏范式，由两个互补阶段组成。首先，我们引入混合原型蒸馏（HPD），它能够对跨模态与模态特有信息实现更可靠的知识迁移。具体而言，HPD 结合主导模态选择，利用高层语义原型执行跨模态语义蒸馏，以降低模态偏差；同时，HPD 执行类内特征变化蒸馏，以保留模态特有的结构细节。其次，为使教师模型逐步产生更均衡、更鲁棒的模态表征，我们让学生模型向教师提供来自非主导模态的反馈，从而使整个蒸馏过程受益。在三个数据集上的实验表明，本方法达到最优鲁棒性（例如在 DeLiVER 上缺失模态性能提升 2.40%），而完整模态性能几乎不下降（mIoU 仅下降 0.1%）。此外，使用不同骨干网络（AnySeg 与 CMNeXt）的评估进一步验证了 RobustSeg 的泛化能力。
英文原文摘要 1490 字符
Multimodal semantic segmentation (MMSS) faces significant challenges in real-world applications due to incomplete, degraded, or missing sensor data. To address this, we propose RobustSeg, an efficient teacher-student framework that enhances model robustness under missing-modality conditions while maintaining strong performance in full-modality scenarios. RobustSeg adopts a feedback-based self-distillation paradigm consisting of two complementary stages. Firstly, we introduce Hybrid Prototype Distillation (HPD), which enables more reliable knowledge transfer of both cross-modal and modality-specific aspects. Concretely, combined with dominant-modality selection, HPD performs cross-modal semantic distillation with high-level semantic prototypes to reduce modality bias. Meanwhile, HPD conducts intra-class feature variation distillation for modality-specific structural details. Secondly, to enable the teacher model to gradually produce more balanced and robust modality representations, we make the student model provide feedback from the non-dominant modality to the teacher, benefiting the entire distillation process. Experiments on three datasets demonstrate that our method achieves state-of-the-art robustness (e.g., +2.40% missing-modality performance on DeLiVER) while causing almost no degradation in full-modality performance (only -0.1% mIoU). Moreover, evaluations using different backbones (AnySeg and CMNeXt) further validate the generalization ability of RobustSeg.