SVHalluc：面向音视频大语言模型的语音-视觉幻觉基准
SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models
为视觉研究者提供了一个诊断音视频大模型「语音-视觉」跨模态对齐能力的评测框架，其语义与时序双维度探针可直接用于检验视频理解模型是否真正将语音内容落地到对应画面，而非仅靠单模态先验作答。
原文依据 we introduce SVHalluc, the first comprehensive benchmark for evaluating speech-vision hallucination in audio-visual LLMs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管音视频大语言模型（LLMs）取得了成功，它们仍会产生看似合理却缺乏依据的输出，即幻觉。现有基准主要关注环境声音（如狗吠）以指示事件是否发生。相比之下，人类语音承载着本质上不同且丰富的语义与时序结构，然而当前模型能否准确地将语音内容与相应的视觉信号对齐，仍未被探索。在本工作中，我们表明语音内容可以诱发音视频 LLMs 的幻觉。为系统研究这一问题，我们提出了 SVHalluc，这是首个用于评估音视频 LLMs 中语音-视觉幻觉的综合性基准。该基准从语义与时序这两个关键且互补的方面诊断语音-视觉幻觉。实验结果表明，最优的开源音视频 LLMs 难以将语音内容与相应的视觉信号对齐，在多项任务上准确率接近随机水平。相比之下，Gemini-2.5 Pro 显著优于开源模型。我们的分析表明，其失败源于跨模态理解能力有限，尽管它们在单模态感知上表现强劲。本工作揭示了当前音视频 LLMs 一个全新且根本性的局限，并凸显了以语音为依托的视频理解的需求。项目主页：https://chenshuang-zhang.github.io/projects/svhalluc/。
英文原文摘要 1476 字符
Despite the success of audio-visual large-language models (LLMs), they can produce plausible but ungrounded outputs, termed hallucination. Existing benchmarks focus on environmental sounds (e.g., dog barking) to indicate event occurrence. In contrast, human speech carries fundamentally different, rich semantics and temporal structures, yet it remains unexplored whether current models can accurately align speech content with corresponding visual signals. In this work, we show that speech content can induce hallucinations in audio-visual LLMs. To systematically study this, we introduce SVHalluc, the first comprehensive benchmark for evaluating speech-vision hallucination in audio-visual LLMs. Our benchmark diagnoses speech-vision hallucinations from two critical and complementary aspects: semantic and temporal. Experimental results demonstrate that state-of-the-art open-source audio-visual LLMs struggle with aligning speech content with corresponding visual signals, with a near-random accuracy on multiple tasks. In contrast, Gemini-2.5 Pro significantly outperforms the open-source models. Our analysis suggests that their failures stem from limited ability in cross-modality understanding, despite strong performance in single-modality perception. Our work uncovers a new and fundamental limitation of current audio-visual LLMs and highlights the need for speech-grounded video comprehension. Project page: https://chenshuang-zhang.github.io/projects/svhalluc/.