HumanVBench：以自动合成基准探测多模态大模型中以人为中心的视频理解能力
HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarks
提供了一套可复用的自动化基准构建流水线：用强模型生成高质量视频标注，并把模型自身的错误系统性地转化为干扰项，从而以极低人工成本构造细粒度多选题评测集；做视频 MLLM 评测或需要构造难负样本的视觉研究者可直接借鉴这一「错误即干扰项」的合成思路。
原文依据 systematically converting model-induced errors into plausible distractors
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论评估多模态大语言模型（MLLMs）在以人为中心的视频理解方面的细微能力仍是一项巨大挑战，因为现有基准往往忽视情感、行为与跨模态对齐的复杂性。我们提出 HumanVBench，一个综合性视频基准，旨在通过 16 项细粒度任务严格探测上述能力。本工作的基石是一种新颖且可扩展的基准构建方法，包含两条自动化流水线，可在极少人工投入下合成高质量视频标注与具有挑战性的多项选择题。通过利用最优模型进行标注，并将模型自身产生的错误系统性地转化为看似合理的干扰项，我们的框架提供了一台可推广的「机器」，用于构建细致的评测套件。我们在 HumanVBench 上对 30 个领先 MLLM 的广泛评估揭示了关键缺陷，尤其是在感知细微情感以及将语音与视觉线索对齐方面，即便是顶尖的专有模型也未能达到人类水平。我们开源 HumanVBench 及其合成流水线，以推动更具社会智能与能力的视频 MLLM 的发展。
英文原文摘要 1213 字符
Evaluating the nuanced human-centric video understanding capabilities of Multimodal Large Language Models (MLLMs) remains a great challenge, as existing benchmarks often overlook the intricacies of emotion, behavior, and cross-modal alignment. We introduce HumanVBench, a comprehensive video benchmark designed to rigorously probe these capabilities across 16 fine-grained tasks. A cornerstone of our work is a novel and scalable benchmark construction methodology, featuring two automated pipelines that synthesize high-quality video annotations and challenging multiple-choice questions with minimal human labor. By leveraging state-of-the-art models for annotation and systematically converting model-induced errors into plausible distractors, our framework provides a generalizable "machine" for creating nuanced evaluation suites. Our extensive evaluation of 30 leading MLLMs on HumanVBench reveals critical deficiencies, particularly in perceiving subtle emotions and aligning speech with visual cues, with even top proprietary models falling short of human performance. We open-source HumanVBench and our synthesis pipelines to catalyze the development of more socially intelligent and capable video MLLMs.