视觉语言模型 · CVPR2026 ·2026.06.03

用视频思考:视频生成作为有前景的多模态推理范式

Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm

Jingqi Tong、Yurong Mou、Hangcheng Li、Mingzhe Li、Yongzhuo Yang、Ming Zhang 等 14 人

与你有关的部分

把视频生成模型(如 Sora-2)当作统一的多模态推理器,用视频帧作为推理媒介,为视觉研究者提供了一个可复用的思路:将生成式视频模型直接用于视觉问答、数学推理等任务,并可用自一致性与上下文学习提升其推理表现。

原文依据 we propose "Thinking with Video", a new paradigm that leverages video generation models such as Sora-2 to use video frames as a unified medium for multimodal reasoning
可迁移到 多模态推理视觉问答视频生成基准评测

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

“用文本思考”与“用图像思考”范式显著提升了大语言模型(LLM)与视觉语言模型(VLM)的推理能力。然而,这些范式存在固有局限:(1)图像仅捕捉单一时刻,无法表征动态过程或连续变化;(2)文本与视觉被割裂为不同模态,阻碍了统一的多模态理解与生成。为此,我们提出“用视频思考”,一种利用 Sora-2 等视频生成模型、以视频帧作为多模态推理统一媒介的新范式。为支撑这一探索,我们构建了 Video Thinking Benchmark(VideoThinkBench),涵盖以视觉为中心的任务(如目测谜题)与以文本为中心的任务(如 GSM8K 与 MMMU)。在 VideoThinkBench 上的评测表明,Sora-2 是一个有能力的推理器。在以视觉为中心的任务上,Sora-2 与最优的 VLM 相当,在目测谜题上甚至超过 GPT-5 达 10%。在以文本为中心的任务上,Sora-2 在 MATH 上取得 92% 的准确率,在 MMMU 上取得 69.2% 的准确率。此外,我们系统分析了这些能力的来源。我们还发现,自一致性与上下文学习能够提升 Sora-2 的表现。总之,我们的发现表明视频生成模型是潜在的统一多模态理解与生成模型,将“用视频思考”定位为一种潜在的统一多模态推理范式。

英文原文摘要 1524 字符

The "Thinking with Text" and "Thinking with Images" paradigms significantly improve the reasoning abilities of large language models (LLMs) and Vision-Language Models (VLMs). However, these paradigms have inherent limitations. (1) Images capture only single moments and fail to represent dynamic processes or continuous changes, and (2) The separation of text and vision as distinct modalities, which hinders unified multimodal understanding and generation. Therefore, we propose "Thinking with Video", a new paradigm that leverages video generation models such as Sora-2 to use video frames as a unified medium for multimodal reasoning. To support this exploration, we developed the Video Thinking Benchmark (VideoThinkBench), which covers both vision-centric tasks (e.g., Eyeballing Puzzles) and text-centric tasks (e.g., GSM8K and MMMU). Our evaluation on VideoThinkBench establishes Sora-2 as a capable reasoner. On vision-centric tasks, Sora-2 is comparable to state-of-the-art (SOTA) VLMs, and even surpasses GPT-5 by 10% on eyeballing puzzles. On text-centric tasks, Sora-2 achieves 92% accuracy on MATH, and 69.2% accuracy on MMMU. Furthermore, we systematically analyze the source of these abilities. We also find that self-consistency and in-context learning can improve Sora-2's performance. In summary, our findings show that the video generation model is the potential unified multimodal understanding and generation model, positioning "Thinking with Video" as a potential unified multimodal reasoning paradigm.

标签

会议 CVPR2026
页码 41121-41129
Bibkey
来源 CVF Open Access