面向多模态社交交互的多说话人注意力对齐
Multi-speaker Attention Alignment for Multimodal Social Interaction
针对多说话人视频场景中视觉 token 与文本 token 缺乏说话人一致对齐的问题，提出无需训练参数、无需改架构的注意力偏置注入方案：先动态选出负责视觉 grounding 的注意力头，再用说话人位置与现有注意力图计算社交感知偏置，直接加进注意力机制。做视频 VLM、多模态 grounding 或注意力分析的工作可直接复用这套「检测头选择 + 位置先验偏置」的免训练增强思路。
原文依据 visual and textual tokens lack speaker-consistent alignment, exhibiting substantially weaker cross-modal attention than in object-centric images
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论理解视频中的社交交互，需要对言语与非言语线索之间的动态相互作用进行推理：谁在说话、对谁说、以及伴随何种目光或手势。尽管多模态大语言模型（MLLMs）是天然的候选方案，但简单地加入视觉输入在社交任务上带来的增益却出人意料地不一致。我们对当前最优 MLLMs 内部跨模态注意力的定量分析揭示了一个核心失效模式：在多说话人场景中，视觉 token 与文本 token 缺乏与说话人一致的对齐，其跨模态注意力显著弱于以物体为中心的图像。为解决这一问题，我们提出一种可集成到现有 MLLMs 中的多模态多说话人注意力对齐方法。首先，我们引入动态跨模态检测头选择，以识别最负责 grounding 的注意力头。随后，将由现有注意力模式与说话人位置计算得到的自适应社交感知注意力偏置注入注意力机制。该偏置强化说话人的视觉表征与其话语之间的对齐，且不引入可训练参数或架构改动。我们将该方法集成到三个不同的 MLLMs（LLaVA-NeXT-Video、Qwen2.5-VL 和 InternVL3）中，并在三个基准（TVQA+、MMSI、OnlineMMSI）上进行评测。在四项社交任务上，结果表明我们的方法提升了 MLLMs 的能力并取得了最优结果。注意力可视化证实，我们的方法成功使模型聚焦于与说话人相关的区域，从而实现更稳健的多方社交推理。
英文原文摘要 1606 字符
Understanding social interaction in video requires reasoning over a dynamic interplay of verbal and non-verbal cues: who is speaking, to whom, and with what gaze or gestures.While Multimodal Large Language Models (MLLMs) are natural candidates, simply adding visual inputs yields surprisingly inconsistent gains on social tasks. Our quantitative analysis of cross-modal attention inside state-of-the-art MLLMs reveals a core failure mode: in multi-speaker scenes, visual and textual tokens lack speaker-consistent alignment, exhibiting substantially weaker cross-modal attention than in object-centric images.To address this, we propose a multimodal multi-speaker attention alignment method that can be integrated into existing MLLMs. First, we introduce dynamic cross-modal head selection to identify attention heads most responsible for grounding. Then, an adaptive social-aware attention bias, computed from existing attention patterns and speaker locations, is injected into the attention mechanism. This bias reinforces alignment between a speaker's visual representation and their utterances without introducing trainable parameters or architectural changes.We integrate our method into three distinct MLLMs (LLaVA-NeXT-Video, Qwen2.5-VL, and InternVL3) and evaluate on three benchmarks (TVQA+, MMSI, OnlineMMSI). Across four social tasks, results demonstrate that our approach improves the ability of MLLMs and achieves state-of-the-art results.Attention visualizations confirm our method successfully focuses the model on speaker-relevant regions, enabling more robust multi-party social reasoning.