面向多模态情感分析的冲突感知自适应交叉重建
Conflict-Aware Adaptive Cross-Reconstruction for Multimodal Sentiment Analysis
对做多模态(视觉-音频-文本)融合的视觉研究者,这篇提供了用「跨模态重建」替代相似度损失来做共享表征隐式对齐的思路,并给出按模态冲突程度自适应加权、抑制冲突模态的具体机制,可直接迁移到视觉主导的多模态融合任务中。
原文依据 It reconstructs each target modality using its own specific features and the shared features of other modalities, achieving implicit alignment of shared representations.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于解耦的方法在学习共享表征时被广泛用于多模态情感分析。然而,现有方法往往忽略同一样本内不同模态之间潜在的情感冲突。它们通常采用模态内重建并依赖相似度损失来对齐共享表征,在存在此类冲突时可能扭曲共享语义。为解决这些问题,我们提出一种冲突感知自适应交叉重建方法(CACR)。首先,我们形式化定义了情感冲突,并设计了一种冲突感知加权策略。该策略为每个模态计算冲突分数,并将其映射为相应的交叉重建权重。其次,我们构建了一个交叉重建模块。该模块利用目标模态自身的特有特征与其他模态的共享特征来重建每个目标模态,从而实现共享表征的隐式对齐。结合上述权重,该模块能够有效抑制冲突模态并缓解语义歧义。此外,我们开发了一个细粒度情感精炼模块,从视觉和音频特有特征中捕捉细粒度线索,以补充文本语义。在三个数据集上的大量实验表明,CACR 优于现有的最优方法,证明了其在处理情感冲突方面的有效性。
英文原文摘要 1404 字符
Disentanglement-based methods for learning shared representations are widely used in multimodal sentiment analysis. However, existing methods often overlook potential emotional conflicts across modalities within the same sample. They typically adopt intra-modal reconstruction and rely on similarity losses to align shared representations, which may distort the shared semantics under such conflicts. To address these, we propose a Conflict-aware Adaptive Cross-Reconstruction approach (CACR). First, we formally define emotional conflict and design a conflict-aware weighting strategy. This strategy calculates conflict scores for each modality and maps them to the corresponding cross-reconstruction weights. Second, we construct a cross-reconstruction module. It reconstructs each target modality using its own specific features and the shared features of other modalities, achieving implicit alignment of shared representations. Coupled with the aforementioned weights, this module effectively suppresses conflicting modalities and mitigates semantic ambiguity. Furthermore, we develop a fine-grained sentiment refinement module that captures fine-grained cues from visual- and audio-specific features to supplement textual semantics. Extensive experiments on three datasets show that CACR outperforms existing state-of-the-art methods, demonstrating its effectiveness in handling emotional conflict.