视频即答案：用联合 GRPO 预测并生成下一个视频事件
Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO
提出 Joint-GRPO 强化学习框架，将视觉语言模型与视频扩散模型联合优化，使 VLM 生成利于可视化的描述，VDM 生成与描述和视觉上下文一致的视频，为多模态推理与生成一体化提供可复用范式。
原文依据 we introduce VANS, a model that leverages reinforcement learning to align a Vision-Language Model (VLM) with a Video Diffusion Model (VDM) for VNEP.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管语言模型已在许多现实应用中产生重要影响，视频生成仍主要局限于娱乐领域。受视频能够展示仅靠语言难以传达的物理世界信息这一固有能力的启发（例如，想象仅用文字教人打领带），我们发现了一个尚未充分利用的机会：将视频扩展为下一事件预测（NEP）的一种新答案模态，并将其形式化为视频下一事件预测（VNEP）。既有的 NEP 任务以视频和程序性或预测性问题为输入，以文本预测下一事件，而 VNEP 要求动态的视频响应。这种从“讲述”到“展示”的转变，为程序性学习和创造性探索解锁了更直观、更个性化的答案。然而，该任务对现有模型仍具挑战性，因为它要求理解多模态输入、进行指令条件推理，并生成视觉与语义一致的视频。为此，我们提出 VANS，一个利用强化学习将视觉语言模型（VLM）与视频扩散模型（VDM）对齐以用于 VNEP 的模型。VANS 的核心是我们提出的联合 GRPO，它协调 VLM 与 VDM 作为一个整体运作。在各自输出上共享奖励的驱动下，它优化 VLM 以生成既准确又利于可视化的描述，同时引导 VDM 生成忠实于这些描述和输入视觉上下文的视频。为支持该学习，我们构建了 VANS-Data-100K，一个专用于 VNEP 任务的数据集。在程序性和预测性基准上的实验表明，VANS 在视频事件预测和可视化两方面均达到最优性能。
英文原文摘要 1779 字符
While language models have become impactful in many real-world applications, video generation remains largely confined to entertainment. Motivated by video's inherent capacity to demonstrate physical-world information that is difficult to convey through language alone (e.g., imagine teaching someone to tie a tie using only text), we identify an underutilized opportunity to extend video as a new answer modality for Next-Event Prediction (NEP), formalized as Video-Next-Event Prediction (VNEP). While the established NEP task takes a video with a procedural or predictive question as input to predict the next event in text, VNEP requires dynamic video responses. This shift from telling to showing unlocks more intuitive and customized answers for procedural learning and creative exploration. However, this task remains challenging for existing models, as it demands an understanding of multimodal input, instruction-conditioned reasoning, and the generation of video with visual and semantic consistency. To address this, we introduce VANS, a model that leverages reinforcement learning to align a Vision-Language Model (VLM) with a Video Diffusion Model (VDM) for VNEP. The core of VANS is our proposed Joint-GRPO that orchestrates the VLM and VDM to function as a unit. Driven by a shared reward on their respective output, it optimizes the VLM to produce captions that are both accurate and friendly to visualize, while guiding the VDM to generate videos that are faithful to these captions and the input visual context. To enable this learning, we craft VANS-Data-100K, a dedicated dataset for the VNEP task. Experiments on procedural and predictive benchmarks demonstrate that VANS achieves state-of-the-art performance in both video event prediction and visualization.