基于聚类的多度量表征学习策略用于细粒度多模态情感分析
Multi-Metric Representation Learning Strategy Based on Clustering for Fine-Grained Multimodal Sentiment Analysis
把多模态表征按情感类别聚类后，在全局与簇内局部两套度量下做投影与决策级融合，这一「聚类+多度量」思路可直接迁移到视觉-语言模型中的细粒度情感/意图判别与跨模态对齐。
原文依据 minimizes distances within similar sentiment pairs while maximizing dissimilar ones, aiming to learn a global metric and local metrics in each cluster
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态情感分析（MSA）旨在通过多模态数据识别人类情感。尽管 MSA 已取得显著进展，我们发现将不同模态的数据整合到同一表征空间时，情感类中心往往相互重叠。本文提出一种基于聚类的多度量表征学习策略（MMRest），通过灵活的多度量表征学习缓解该问题，使模型能够学习细粒度情感。具体而言，我们首先设计了一个名为聚类上的多度量多模态学习（MMC）的模块，它最小化相似情感对之间的距离、最大化不相似情感对之间的距离，旨在从多模态数据中学习一个全局度量和每个簇内的局部度量。随后，我们开发了投影与决策级融合（PDLF）模块，包含两部分：一部分利用最优的全局与局部度量获得投影值；另一部分将该投影值与通过融合单模态和多模态表征得到的中间分数相结合，得到最终的情感预测分数。在 CMU-MOSI 和 CMU-MOSEI 数据集上的大量实验表明，该方法通过有效学习细粒度情感边界，在多项评价指标和参数量上均显著优于最优方法。若论文被接收，代码将开源。
英文原文摘要 1482 字符
Multimodal sentiment analysis (MSA) aims to identify human emotions through multimodal data. Despite considerable advances in MSA, we find that emotional class centers often overlap when integrating data from different modalities into the same representation space. In this paper, we propose a novel Multi-Metric Representation learning strategy based on clustering (MMRest) to alleviate this issue through flexible multi-metric representation learning, enabling the model to learn fine-grained sentiments. Specifically, we first design a module termed Multi-metric Multimodal learning on Clusters (MMC), which minimizes distances within similar sentiment pairs while maximizing dissimilar ones, aiming to learn a global metric and local metrics in each cluster from multimodal data. Afterwards, we develop a Projection and Decision-Level Fusion (PDLF) module, including two parts. One part utilizes the optimal global and local metrics to obtain a projection value. The other part combines the projection value with an intermediate score which is obtained through the fusion of unimodal and multimodal representations to obtain the final sentiment prediction score. Extensive experiments on the CMU-MOSI and CMU-MOSEI datasets demonstrate that our method is significantly superior to state-of-the-art methods on various evaluation indicators and parameter count, by effectively learning fine-grained emotional boundaries. The code will be made open-source if the paper is accepted.