听见你在哪：用视觉与空间声音教大语言模型空间推理
Hear you are: Teaching LLMs Spatial Reasoning with Vision and Spatial Sound
把双耳空间音频作为额外输入通道接入大语言模型，为视觉问答提供声源方位线索；当语义对应缺失或存在多个语义匹配时，空间对齐可作为消歧依据，这一设定可直接迁移到机器人听觉定位、视频中的声源定位与多模态推理任务。
原文依据 We design a spatial audio-visual question answering dataset to cover scenarios where semantic correspondence between audio and visual signals is absent but spatial alignment exists
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论许多视听学习方法关注音频与视觉信息的对齐，或基于语义对应，或基于时序对应。然而，这些工作大多使用单声道音频，其中并不包含声源空间位置的信息。与之相反，人类和其他动物利用双耳听觉来感知这种空间信息。将空间声音与视觉感知相结合，能够支撑强大的高层推理：例如，一个人寻找手机时可能听到铃声从放在桌上的背包中传出，从而迅速推断出丢失的手机就在背包里。本文研究视听空间推理问题。我们设计了一个空间视听问答数据集，覆盖音频与视觉信号之间不存在语义对应但存在空间对齐的场景，以及存在多个视听语义对应、需要借助空间推理进行消歧的情形。我们提出一个模型，通过大语言模型将音频与视觉两种模态连接起来，从而学习跨模态的空间理解，并通过实验证明空间声音感知是本任务不可或缺的组成部分。
英文原文摘要 1260 字符
Many audio-visual learning methods have focused on aligning audio and visual information, either through semantic or temporal correspondence. However, most of these works have utilized monaural audio, which does not contain information about the spatial location of the sound source. In contrast, humans and other animals utilize binaural hearing to perceive this spatial information. Combining spatial sound and visual perception enables powerful high-level reasoning: for example, a person looking for their phone may hear the ringing sound coming from a backpack sitting on a table, and quickly infer that the missing phone is inside the backpack. In this paper, we investigate the problem of Audio-Visual Spatial Reasoning. We design a spatial audio-visual question answering dataset to cover scenarios where semantic correspondence between audio and visual signals is absent but spatial alignment exists, as well as cases with multiple audio-visual semantic correspondences that require spatial reasoning to disambiguate. We propose a model that learns spatial comprehension across the audio and vision modalities by connecting them with a large language model and experimentally demonstrate that spatial sound perception is an essential part of our task.