OneStory:基于自适应记忆的连贯多镜头视频生成
OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory
将多镜头视频生成重构为逐镜头自回归的下一镜头生成任务,并复用预训练 I2V 模型;其帧选择构建全局记忆与重要性引导的 patch 化压缩上下文,可直接迁移到长视频生成、故事化叙事以及任何需要跨片段长程一致性的条件生成任务。
原文依据 OneStory reformulates MSV as a next-shot generation task, enabling shot-by-shot autoregressive synthesis
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论真实世界视频中的叙事往往通过多个镜头展开——这些片段不连续却在语义上相互关联,共同传达连贯的叙事。然而,现有多镜头视频生成(MSV)方法难以有效建模长程跨镜头上下文,因为它们依赖有限的时间窗口或单一关键帧条件,在复杂叙事下性能下降。在本工作中,我们提出 OneStory,实现全局而紧凑的跨镜头上下文建模,从而生成一致且可扩展的叙事。OneStory 将 MSV 重构为下一镜头生成任务,实现逐镜头的自回归合成,同时利用预训练的图像到视频(I2V)模型提供强视觉条件。我们引入两个关键模块:帧选择模块,基于先前镜头中信息量丰富的帧构建语义相关的全局记忆;自适应条件器,执行重要性引导的 patch 化,生成紧凑上下文以直接作为条件。我们进一步构建了一个高质量多镜头数据集,配有指代性描述,以反映真实世界的叙事模式,并设计了下一镜头范式下的有效训练策略。在我们构建的 60K 数据集上对预训练 I2V 模型进行微调后,OneStory 在文本条件与图像条件两种设定下,于多样且复杂的场景中实现了最优的叙事连贯性,支持可控且沉浸式的长视频叙事。更多细节请参见我们的项目主页。
英文原文摘要 1531 字符
Storytelling in real-world videos often unfolds through multiple shots--discontinuous yet semantically connected clips that together convey a coherent narrative. However, existing multi-shot video generation (MSV) methods struggle to effectively model long-range cross-shot context, as they rely on limited temporal windows or single keyframe conditioning, leading to degraded performance under complex narratives. In this work, we propose OneStory, achieving global yet compact cross-shot context modeling for consistent and scalable narrative generation. OneStory reformulates MSV as a next-shot generation task, enabling shot-by-shot autoregressive synthesis while leveraging pretrained image-to-video (I2V) models for strong visual conditioning. We introduce two key modules: a Frame Selection module that constructs a semantically-relevant global memory based on informative frames from prior shots, and an Adaptive Conditioner that performs importance-guided patchification to generate compact context for direct conditioning. We further curate a high-quality multi-shot dataset with referential captions to mirror real-world storytelling patterns, and design effective training strategies under the next-shot paradigm. Finetuned from a pretrained I2V model on our curated 60K dataset, OneStory achieves state-of-the-art narrative coherence across diverse and complex scenes in both text- and image-conditioned settings, enabling controllable and immersive long-form video storytelling. See our Project Page for more details.