EgoSound：第一人称视频中声音理解基准
EgoSound: Benchmarking Sound Understanding in Egocentric Videos
为视觉研究者提供了一个可复用的第一人称音视频问答基准与七任务分类体系，可用于诊断多模态大模型在离屏事件、空间定位与因果推理上的盲区，并作为音频-视觉联合训练或评测的下游测试床。
原文依据 we introduce EgoSound, the first benchmark designed to systematically evaluate egocentric sound understanding in MLLMs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）近期在视觉-语言理解方面取得了显著进展。然而，人类感知本质上是多感官的，会整合视觉、声音与运动来推理世界。在这些模态中，声音提供了关于空间布局、离屏事件与因果交互的不可或缺的线索，在第一人称场景中尤为如此，因为听觉与视觉信号紧密耦合。为此，我们提出 EgoSound，这是首个用于系统评估 MLLMs 第一人称声音理解的基准。EgoSound 统一了来自 Ego4D 与 EgoBlind 的数据，涵盖有视觉与依赖声音两类体验。它定义了一个包含七项任务的分类体系，覆盖内在声音感知、空间定位、因果推断与跨模态推理。EgoSound 通过多阶段自动生成流程构建，包含 900 个视频上的 7315 个经过验证的问答对。在九个最优 MLLMs 上的综合实验表明，当前模型展现出初步的听觉推理能力，但在细粒度空间与因果理解方面仍然有限。EgoSound 为推进多感官第一人称智能、弥合看见与真正听见世界之间的差距奠定了具有挑战性的基础。项目主页：https://groolegend.github.io/EgoSound/。
英文原文摘要 1355 字符
Multimodal Large Language Models (MLLMs) have recently achieved remarkable progress in vision-language understanding. Yet, human perception is inherently multisensory, integrating sight, sound, and motion to reason about the world. Among these modalities, sound provides indispensable cues about spatial layout, off-screen events, and causal interactions, particularly in egocentric settings where auditory and visual signals are tightly coupled. To this end, we introduce EgoSound, the first benchmark designed to systematically evaluate egocentric sound understanding in MLLMs. EgoSound unifies data from Ego4D and EgoBlind, encompassing both sighted and sound-dependent experiences. It defines a seven-task taxonomy spanning intrinsic sound perception, spatial localization, causal inference, and cross-modal reasoning. Constructed through a multi-stage auto-generative pipeline, EgoSound contains 7315 validated QA pairs across 900 videos. Comprehensive experiments on nine state-of-the-art MLLMs reveal that current models exhibit emerging auditory reasoning abilities but remain limited in fine-grained spatial and causal understanding. EgoSound establishes a challenging foundation for advancing multisensory egocentric intelligence, bridging the gap between seeing and truly hearing the world. Project page: https://groolegend.github.io/EgoSound/.