MovieRecapsQA：一个多模态开放式视频问答基准
MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering Benchmark
对做视频理解或多模态大模型的人，这里提供了一个可复用的开放式视频问答评测协议：用「答案应包含的事实」构造无参考指标，并可在全长电影、完整回顾视频、对齐片段等不同视觉输入设定下对比模型，从而把视觉感知与视觉推理的瓶颈分开诊断。
原文依据 the primary bottleneck is visual perception, not visual reasoning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论理解电影等真实世界视频需要整合视觉与对话线索。然而，现有的视频问答基准难以刻画这种多模态推理，且由于自由形式答案难以评估，大多退化为简单的多项选择题。我们提出了一个新的开放式多模态视频问答基准 MovieRecapsQA，它基于电影回顾视频构建——这是一类独特的 YouTube 内容，通过旁白描述电影中的关键片段来概括整部影片（回顾视频）。我们从 60 个回顾视频的旁白转写文本（回顾摘要）中生成 8.2K 个问题，以及每个答案中预期应包含的必要「事实」；前者有助于构造需要多模态推理的问题，后者则支持构建可应用于开放式回答的无参考评估指标。据我们所知，这是首个无参考的开放式视频问答基准。该基准允许每个问题在不同的输入视频设定下进行评估：给定 (a) 全长电影，(b) 完整的（11 分钟）回顾视频（仅视觉），(c) 14 分钟的对齐电影场景，即与问题相关的电影场景，以及 (d) 1.2 分钟的对齐回顾视频场景。在所有情况下，都提供任何相关电影对话的文本。每个问题按其回答所需的模态——视觉、对话或两者兼有——进行分类，从而支持对多模态能力的细粒度评估。我们在设定 (d) 下对七个最优多模态大模型进行了基准测试，发现：(i) 只有我们的无参考指标能产生有意义的、与人类判断一致的模型区分度；(ii) 以视觉为中心的问题在所有模型上得分最低；(iii) 移除视觉输入通常会提升模型的事实性；(iv) 主要瓶颈在于视觉感知，而非视觉推理。
英文原文摘要 1853 字符
Understanding real-world videos such as movies requires integrating visual and dialogue cues. Yet existing VideoQA benchmarks struggle to capture this multimodal reasoning and, given the difficulty of evaluating free-form answers, largely resort to simple multiple choice questions. We introduce a novel open-ended multimodal VideoQA benchmark, MovieRecapsQA, created using movie recap videos -- a distinctive type of YouTube content that summarizes a film via a voiceover description of key clips from the movie (recap video). From the transcribed voiceover (recap summary) of 60 recap videos, we generate 8.2K questions along with the necessary "facts" expected in each answer; the former facilitates the creation of questions that require mutimodal reasoning and the latter allow the construction of a reference-free evaluation metric that can be applied to open-ended responses. To our knowledge, this is the first reference-free open-ended VideoQA benchmark. The benchmark allows each question to be evaluated in different input video settings: given (a) the full-length movie, (b) the full ( 11 min) recap video (visual only), (c) 14 min of aligned movie scenes, i.e, movie scenes relevant to the question, and (d) 1.2 min of aligned recap video scenes. In all cases, the text of any associated movie dialogue is provided. Each question is categorized by the modality required to answer it---visual, dialogue, or both---enabling fine-grained evaluation of multimodal capabilities. We benchmark (setting (d)) seven state-of-the-art MLLMs and find that (i) only our reference-free metric produces meaningful human-aligned model separation; (ii) vision-centric questions yield the lowest scores across all models; (iii) removing visual input often improves model factuality; and (iv) the primary bottleneck is visual perception, not visual reasoning.