HAVE-Bench：从感知到交互的层级化音视频评测基准
HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction
做音视频多模态的视觉研究者可把 HAVE-Bench 的三级认知层级（感知-推理-交互）与多轮交互标注协议当作评测模板，用来诊断模型在语音驱动视觉问答上相对文本-图像设定的性能落差，并据此设计跨模态推理能力迁移的训练目标。
原文依据 speech-driven visual question answering (VQA) performance significantly lagging behind the text-image setting
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）已从视觉-语言系统扩展到包含音频，开启了跨模态推理与交互的新能力。针对现有基准主要聚焦感知任务、缺乏统一认知评测框架的局限，我们提出层级化音视频评测基准（HAVE-Bench）。该基准沿感知、推理、交互三级认知层级系统评估 MLLMs 的音频相关能力，使用 2,451 条精选样本以及人工标注的多轮交互级任务。基于这一统一框架的实验揭示了现有模型在推理与交互层级上的显著差距，其中语音驱动的视觉问答（VQA）性能明显落后于文本-图像设定。这些发现凸显了增强模型处理长时复杂音频、并促进推理能力从视觉-文本域向音视频域迁移的紧迫性。
英文原文摘要 1057 字符
Multimodal large language models (MLLMs) have expanded from vision-language systems to include audio, unlocking new capabilities in cross-modal reasoning and interaction. To address the limitation that existing benchmarks focus mainly on perception tasks and lack a unified cognitive evaluation framework, we propose Hierarchical Audio-Visual Evaluation Benchmark (HAVE-Bench). It systematically evaluates the audio-related capabilities of MLLMs along a three-level cognitive hierarchy: Perception, Reasoning, and Interaction, utilizing 2,451 curated samples and manually annotated multi-turn interaction-level tasks. Experiments using this unified framework reveal significant gaps in existing models at the reasoning and interaction levels, with speech-driven visual question answering (VQA) performance significantly lagging behind the text-image setting. These findings underscore the urgency of enhancing models' handling of long and complex audio and facilitating the transfer of reasoning capabilities from the vision-text to the audio-visual domain.