帧链：通过帧感知推理提升多模态大语言模型的视频理解能力
Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning
提出单阶段帧感知推理链（Chain-of-Frames），让视频LLM在推理中显式引用关键帧，无需辅助模块即可提升时序定位与视频理解性能；其构建的CoF-Data数据集和合成数据增强策略可复用于视频推理任务。
原文依据 we propose the first video LLMs with single-stage reasoning that includes explicit references to relevant frames
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期研究表明，引导大语言模型（LLM）在回答用户请求前用自然语言生成推理轨迹，能显著提升其跨任务表现。这一方法已扩展至多模态LLM，使模型能对输入图像和视频内容生成思维链（CoT）。针对视频输入，先前工作或采用复杂多步流程从视频中提取相关帧并纳入CoT，或生成更简单的单阶段推理轨迹但牺牲了时序定位能力。本文提出首个具备单阶段推理的视频LLM，其推理过程显式引用相关帧，从而减少推理过程中的时序不一致性。该方法简单、统一且自包含，采用单阶段推理处理复杂视频理解任务，不依赖帧选择或字幕生成的辅助模块。为此，我们首先构建了CoF-Data——一个大规模数据集，包含来自自然视频和合成视频的多样化问题、答案及对应的帧级推理轨迹，覆盖多种主题与任务。通过在此帧链（CoF）数据上微调视频LLM得到的模型，能生成准确识别关键帧以回答问题的推理轨迹，从而在多个视频理解基准上持续提升性能。令人惊讶的是，我们发现仅使用合成数据——尽管相对于这些真实世界基准属于分布外数据——也能显著提升模型准确率。
英文原文摘要 1632 字符
Recent work has shown that eliciting Large Language Models (LLMs) to generate reasoning traces in natural language before answering the user's request can significantly improve their performance across tasks. This approach has been extended to multimodal LLMs, where the models can produce chains-of-thoughts (CoT) about the content of input images and videos. For video inputs, prior works use complex multi-step pipelines that extract and include relevant frames from videos in the CoT, or produce simpler single-stage reasoning traces at the expense of poor temporal grounding. Here, we propose the first video LLMs with single-stage reasoning that includes explicit references to relevant frames, thereby reducing temporal inconsistencies in the reasoning process. Our approach is simple, unified, and self-contained, employing a single-stage inference to handle complex video understanding tasks without relying on auxiliary modules for frame selection or caption generation. For this, we first create CoF-Data, a large dataset of diverse questions, answers, and corresponding frame-grounded reasoning traces from both natural and synthetic videos, spanning various topics and tasks. Our models, obtained by fine-tuning video LLMs on this chain-of-frames (CoF) data, generate reasoning traces that accurately identify key frames to answer given questions. In turn, this consistently improves performance across multiple video understanding benchmarks. Surprisingly, we find that synthetic data alone, despite being out-of-distribution with respect to these real-world benchmarks, provides a significant boost in model accuracy.