面向不配对测试时分布外文本-图像到视频生成的推理扩散模型
Reasoning Diffusion for Unpaired Test Time Out-of-distribution Text-Image to Video Generation
ReasonDiff 利用多模态 LLM 为不配对文本-图像输入生成逐帧叙事，并通过 AlignFormer 的多阶段时序锚框注意力预测帧级潜变量，为视频扩散模型提供结构化引导，可复用于需要跨模态时序推理的视频生成任务。
原文依据 ReasonDiff designs a VisionNarrator module to harness the powerful reasoning abilities of a multi-modal LLM to analyze the unpaired text-image inputs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论文本-图像到视频生成旨在根据给定的文本-图像输入合成视频。然而，现有方法通常假设输入文本和图像所携带的语义信息是完美配对且时序对齐的，并同时出现在生成的视频中。因此，现有文献难以处理分布外（OOD）的“不配对”文本-图像输入，而在更普遍和现实的场景中，i）文本和图像携带的语义信息可能出现在不同的时间戳，且ii）条件图像可以出现在任意位置，而不是合成视频的第一帧。这种OOD设置下的视频生成迫切需要对给定文本描述和所指图像之间的内在联系进行推理，这具有挑战性且尚未被探索。为应对这一挑战，本文首次研究了不配对文本-图像到视频生成问题，提出了ReasonDiff，一种从非配对文本-图像输入准确生成视频的新模型。具体而言，ReasonDiff设计了一个VisionNarrator模块，利用多模态LLM的强大推理能力分析非配对文本-图像输入，生成连贯的逐帧叙事，从而在时间上对齐它们。基于该VisionNarrator模块，ReasonDiff进一步引入了一种新颖的AlignFormer模块，该模块采用多阶段时序锚框注意力机制来预测逐帧潜变量表示。这些推理增强的潜变量随后与条件帧融合，在整个视频生成过程中提供结构化引导。大量实验和消融研究表明，在非配对文本-图像输入下，ReasonDiff在视频生成质量方面优于最优基线。
英文原文摘要 1886 字符
Text-image to video generation aims to synthesize a video conditioned on the given text-image inputs. Nevertheless, existing methods generally assume that the semantic information carried in the input text and image tends to be perfectly paired and temporally aligned, occurring simultaneously in the generated video. As such, existing literature struggles with out-of-distribution (OOD) "unpaired" text-image inputs in the more universal and realistic scenario where i) the semantic information carried by the text and image may occur at different timestamps and ii) the condition image can appear at an arbitrary position rather than the first frame of the synthesized video. Video generation under this OOD setting poses an urgent need to conduct reasoning over the intrinsic connections between the given textual description and referred image, which is challenging and remains unexplored. To address the challenge, in this paper we study the problem of unpaired text-image to video generation for the first time, proposing ReasonDiff, a novel model for accurate video generation from unpaired text-image inputs. Specifically, ReasonDiff designs a VisionNarrator module to harness the powerful reasoning abilities of a multi-modal LLM to analyze the unpaired text-image inputs, producing coherent per-frame narratives that temporally align them. Building upon this VisionNarrator module, ReasonDiff further introduces a novel AlignFormer module, which employs a Multi-stage Temporal Anchor Attention mechanism to predict frame-wise latent representations. These reasoning-enhanced latents are subsequently fused with the condition frame, providing structured guidance throughout the video generation process. Extensive experiments and ablation studies demonstrate that ReasonDiff beats state-of-the-art baselines in terms of video generation quality with unpaired text-image inputs.