Omni-MMSI：面向身份归属的社交交互理解
Omni-MMSI: Toward Identity-attributed Social Interaction Understanding
论文提出从原始音视频与语音中联合感知「谁说了什么、指向谁」的身份归属线索，并在此基础上做思维链社交推理；对视觉研究者而言，可复用其参与者级参考对构建方式与参考引导的多模态推理流程，用于需要身份绑定的多人视频理解任务。
原文依据 we propose Omni-MMSI-R, a reference-guided pipeline that produces identity-attributed social cues with tools and conducts chain-of-thought social reasoning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出 Omni-MMSI，这是一个新任务，要求从原始音频、视觉和语音输入中实现全面的社交交互理解。该任务涉及感知具有身份归属的社交线索（例如，谁在说什么），并对社交交互进行推理（例如，说话者指的是谁）。这一任务对于开发能够感知并回应人类交互的 AI 助手至关重要。与以往在经 oracle 预处理后的社交线索上开展的研究不同，Omni-MMSI 反映的是 AI 助手必须从原始数据进行感知与推理的真实场景。然而，现有流程与多模态大语言模型在 Omni-MMSI 上表现不佳，因为它们缺乏可靠的身份归属能力，从而导致社交交互理解不准确。为应对这一挑战，我们提出 Omni-MMSI-R，这是一种参考引导的流程，它借助工具生成具有身份归属的社交线索，并开展思维链社交推理。为支撑该流程，我们在现有数据集之上构建了参与者级参考对，并整理了推理标注。实验表明，Omni-MMSI-R 在 Omni-MMSI 上优于先进的大语言模型与同类方法。项目主页：https://sampson-lee.github.io/omni-mmsi-project-page。
英文原文摘要 1271 字符
We introduce Omni-MMSI, a new task that requires comprehensive social interaction understanding from raw audio, vision, and speech input. The task involves perceiving identity-attributed social cues (e.g., who is speaking what) and reasoning about the social interaction (e.g., whom the speaker refers to). This task is essential for developing AI assistants that can perceive and respond to human interactions. Unlike prior studies that operate on oracle-preprocessed social cues, Omni-MMSI reflects realistic scenarios where AI assistants must perceive and reason from raw data. However, existing pipelines and multi-modal LLMs perform poorly on Omni-MMSI because they lack reliable identity attribution capabilities, which leads to inaccurate social interaction understanding. To address this challenge, we propose Omni-MMSI-R, a reference-guided pipeline that produces identity-attributed social cues with tools and conducts chain-of-thought social reasoning. To facilitate this pipeline, we construct participant-level reference pairs and curate reasoning annotations on top of the existing datasets. Experiments demonstrate that Omni-MMSI-R outperforms advanced LLMs and counterparts on Omni-MMSI. Project page: https://sampson-lee.github.io/omni-mmsi-project-page.