Narrative Weaver:面向多模态条件下的可控长程视觉一致性
Narrative Weaver: Towards Controllable Long-Range Visual Consistency with Multi-Modal Conditioning
动态记忆库(Memory Bank)机制为长序列生成中的视觉漂移问题提供了可复用的思路,可迁移到视频生成、漫画/分镜生成等需要跨场景一致性的任务;其多阶段渐进训练策略也展示了在有限数据下复用预训练模型做细粒度控制的方法。
原文依据 a novel fine-grained control module featuring a dynamic Memory Bank that prevents visual drift
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文提出 Narrative Weaver,一个解决生成式人工智能中一项根本性挑战的新框架:实现可控、长程且一致的视觉内容生成。现有模型虽擅长生成高保真短篇视觉内容,却难以在长序列中保持叙事连贯性与视觉一致性——这对电影制作和电商广告等真实应用而言是严重局限。Narrative Weaver 首次提出整体性解决方案,将三项关键能力无缝整合:细粒度控制、自动叙事规划与长程一致性。我们的架构将用于高层叙事规划的多模态大语言模型(MLLM)与一个带有动态记忆库的新型细粒度控制模块相结合,后者可防止视觉漂移。为便于实际部署,我们设计了一种渐进式多阶段训练策略,高效利用现有预训练模型,即使在训练数据有限的情况下也能达到最优性能。鉴于缺乏合适的评测基准,我们构建并发布了电商广告视频故事板数据集(EAVSD)——该任务首个综合性数据集,包含超过 33 万张高质量图像及丰富的叙事标注。通过在三种不同场景(可控多场景生成、自主叙事与电商广告)下的广泛实验,我们证明了本方法的优越性,并为 AI 驱动的内容创作开辟了新的可能。
英文原文摘要 1587 字符
We present Narrative Weaver, a novel framework that addresses a fundamental challenge in generative AI: achieving controllable, long-range, and consistent visual content generation. While existing models excel at generating high-fidelity short-form visual content, they struggle to maintain narrative coherence and visual consistency across extended sequences--a critical limitation for real-world applications such as filmmaking and e-commerce advertising.Narrative Weaver introduces the first holistic solution that seamlessly integrates three essential capabilities: fine-grained control, automatic narrative planning, and long-range coherence. Our architecture combines a Multimodal Large Language Model (MLLM) for high-level narrative planning with a novel fine-grained control module featuring a dynamic Memory Bank that prevents visual drift.To enable practical deployment, we develop a progressive, multi-stage training strategy that efficiently leverages existing pre-trained models, achieving state-of-the-art performance even with limited training data. Recognizing the absence of suitable evaluation benchmarks, we construct and release the E-commerce Advertising Video Storyboard Dataset (EAVSD)--the first comprehensive dataset for this task, containing over 330K high-quality images with rich narrative annotations.Through extensive experiments across three distinct scenarios (controllable multi-scene generation, autonomous storytelling, and e-commerce advertising), we demonstrate our method's superiority while opening new possibilities for AI-driven content creation.