原型即提示:多模态情感原型赋予大语言模型执行多模态情感分析的能力
Prototype-as-Prompt: Multimodal Sentiment Prototypes Endowing Large Language Models the Capability to Perform Multimodal Sentiment Analysis
将视觉-音频特征压缩为可解释的多模态情感原型,并作为软提示注入LLM,为视觉-语言模型提供了一种参数高效且语义明确的条件控制方式,可复用于其他多模态理解任务。
原文依据 we propose a prototype-as-prompt framework that maps audio-visual representations into a fixed set of multimodal sentiment prototypes
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态情感分析(MSA)旨在整合文本、声学和视觉信息来预测情感极性。随着大语言模型(LLM)的出现,现有研究通常采用可学习查询来压缩视听表示,并将其作为软提示输入LLM以进行MSA。然而,由于可学习查询的隐式学习机制,这些查询缺乏关于每个查询如何编码情感语义的显式指导。为解决此问题,我们提出了一种原型即提示框架,将视听表示映射到一组固定的多模态情感原型。这些原型随后被用作软提示,指导LLM执行MSA。具体而言,我们首先使用基于重采样的策略将文本和非文本特征压缩为多模态原型。我们进一步引入了情感感知原型学习,显式地将多模态原型与情感语义绑定。为确保多模态情感原型的跨模态一致性和模态内多样性,我们设计了跨模态原型对齐约束和距离加权原型多样性约束。在三个LLM和四个基准数据集上的大量实验表明,PaP仅使用0.09%-0.26%的可训练参数就实现了优越的性能,突显了其有效性和参数效率。
英文原文摘要 1421 字符
Multimodal Sentiment Analysis (MSA) aims to integrate textual, acoustic, and visual information to predict sentiment polarity. With the emergence of Large Language Models (LLMs), existing studies commonly employ learnable queries to compress audio-visual representations and feed them as soft prompts into LLMs for MSA. However, due to the implicit learning mechanism of the learnable queries, these learnable queries lack explicit guidance regarding how each query encodes sentiment semantics. To address this issue, we propose a prototype-as-prompt framework that maps audio-visual representations into a fixed set of multimodal sentiment prototypes. These prototypes are then used as soft prompts to guide the LLM in performing MSA. Concretely, we first compress both textual and non-textual features into multimodal prototypes using a resampling-based strategy. We further introduce a sentiment-aware prototype learning that explicitly binds multimodal prototypes with sentiment semantics. To ensure both cross-modal consistency and intra-modal diversity of multimodal sentiment prototypes, we design a cross-modal prototype alignment constraint and a distance-weighted prototype diversity constraint. Extensive experiments across three LLMs and four benchmark datasets show that PaP achieves superior performance with only 0.09%-0.26% of trainable parameters, highlighting its effectiveness and parameter efficiency.