先增强后平衡的模态协作框架用于鲁棒多模态情感分析
Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
对做多模态融合的视觉研究者，这篇提供了处理「视觉/音频弱模态被文本主导」的可复用思路：用能量势建模各模态贡献并做隐式梯度重平衡，再用样本级可靠性估计自适应调融合权重，可直接迁移到视觉-语言融合、缺失模态鲁棒性等场景。
原文依据 dominant modalities tend to overshadow non-verbal ones, which not only limits their contribution but also induces modality competition during training
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态情感分析（MSA）旨在通过整合文本、音频与视觉模态的异构信号来推断人类情绪。尽管近期方法尝试利用跨模态互补性，但往往难以充分利用较弱的模态。在实践中，各模态的表达能力本身就不平衡：主导模态倾向于压制非语言模态，这不仅限制了后者的贡献，还在训练过程中引发模态竞争。这种不平衡导致融合性能下降，并在模态含噪或缺失时鲁棒性变差。为应对这些挑战，我们提出一种新模型——先增强后平衡的模态协作框架（EBMC）。EBMC 首先通过模态语义解耦（MSD）与跨模态互补增强（CCE）提升表征质量，利用其他模态的信息强化较弱模态。为防止主导模态在联合优化中压制其他模态，EBMC 引入能量引导的模态协调（EMC）机制，通过能量势建模各模态贡献，并借助可微的均衡目标实现隐式梯度重平衡。此外，实例感知的模态信任蒸馏（IMTD）模块估计样本级的模态可靠性并自适应调节融合权重，从而确保对噪声与模态不完整的鲁棒性。在多个 MSA 基准上的大量实验表明，EBMC 达到最优或具有竞争力的结果。此外，EBMC 在模态缺失设定下仍保持强劲性能，凸显了其有效性与鲁棒性。
英文原文摘要 1700 字符
Multimodal sentiment analysis (MSA) seeks to infer human emotions by integrating heterogeneous signals from text, audio, and visual modalities.Although recent approaches attempt to leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities.In practice, the expressive power across modalities is inherently imbalanced: dominant modalities tend to overshadow non-verbal ones, which not only limits their contribution but also induces modality competition during training.This imbalance leads to degraded fusion performance and poor robustness under noisy or missing modalities.To address these challenges, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC first improves representational quality via modality semantic disentanglement (MSD) and cross-modal complementary enhancement (CCE), which strengthens weaker modalities using information from other modalities.To prevent dominant modalities from overwhelming others during joint optimization, EBMC introduces an Energy-guided Modality Coordination (EMC) mechanism that models modality contributions via energy potentials and achieves implicit gradient rebalancing through a differentiable equilibrium objective.Further, an Instance-aware Modality Trust Distillation (IMTD) module estimates sample-level modality reliability and adaptively modulates fusion weights, ensuring robustness against noise and modality incompleteness.Extensive experiments on multiple MSA benchmarks demonstrate that EBMC achieves state-of-the-art or competitive results.Moreover, EBMC maintains strong performance under missing-modality settings, highlighting its effectiveness and robustness.