Video-LLM 如何回答视频问题的实证研究
An Empirical Study on How Video-LLMs Answer Video Questions
用注意力敲除（temporal/spatial/language-to-video）逐层分析 Video-LLM，揭示视频信息抽取集中在浅层、中间层是关键离群层、时空建模更依赖语言引导的检索而非视频 token 自注意力；这些结论可直接用于剪掉冗余的视频自注意力计算，为 Video-LLM 的推理加速和结构设计提供依据。
原文依据 spatial-temporal modeling relies more on language-guided retrieval than on intra- and inter-frame self-attention among video tokens
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论借助大规模数据和预训练语言模型，视频大语言模型（Video-LLM）在回答视频问题方面展现出强大能力。然而，现有工作大多聚焦于提升性能，对理解其内部机制的关注有限。本文旨在通过系统的实证研究填补这一空白。为解释现有 Video-LLM，我们采用注意力敲除作为主要分析工具，并设计三种变体：视频时序敲除、视频空间敲除和语言到视频敲除。随后，我们在不同数量的层（层窗口）上应用这三种敲除。通过仔细控制层窗口和敲除类型，我们提供两种设置：全局设置和细粒度设置。我们的研究揭示了三个关键发现：（1）全局设置表明视频信息抽取主要发生在浅层，形成一个清晰的两阶段过程——低层聚焦于感知编码，而高层处理抽象推理；（2）在细粒度设置中，某些中间层对视频问答产生过大影响，充当关键离群层，而大多数其他层贡献极小；（3）在两种设置中，我们观察到时空建模更依赖语言引导的检索，而非视频 token 之间的帧内和帧间自注意力，尽管后者的计算成本很高。最后，我们证明这些洞见可用于减少 Video-LLM 中的注意力计算。据我们所知，这是首个系统揭示 Video-LLM 内部如何处理和理解视频内容的工作，为未来研究提供了可解释性和效率视角。
英文原文摘要 1780 字符
Taking advantage of large-scale data and pretrained language models, Video Large Language Models (Video-LLMs) have shown strong capabilities in answering video questions. However, most existing efforts focus on improving performance, with limited attention to understanding their internal mechanisms. This paper aims to bridge this gap through a systematic empirical study. To interpret existing VideoLLMs, we adopt attention knockouts as our primary analytical tool and design three variants: Video Temporal Knockout, Video Spatial Knockout, and Language-to-Video Knockout. Then, we apply these three knockouts on different numbers of layers (window of layers). By carefully controlling the window of layers and types of knockouts, we provide two settings: a global setting and a fine-grained setting. Our study reveals three key findings: (1) Global setting indicates Video information extraction primarily occurs in early layers, forming a clear two-stage process--lower layers focus on perceptual encoding, while higher layers handle abstract reasoning; (2) In the fine-grained setting, certain intermediate layers exert an outsized impact on video question answering, acting as critical outliers, whereas most other layers contribute minimally; (3) In both settings, we observe that spatial-temporal modeling relies more on language-guided retrieval than on intra- and inter-frame self-attention among video tokens, despite the latter's high computational cost. Finally, we demonstrate that these insights can be leveraged to reduce attention computation in Video-LLMs. To our knowledge, this is the first work to systematically uncover how Video-LLMs internally process and understand video content, offering interpretability and efficiency perspectives for future research.