生成模型与扩散 · CVPR2026 ·2026.06.03

Wan-Weaver:通过解耦训练实现交错多模态生成

Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training

Jinbo Xing、Zeyinzi Jiang、Yuxiang Tuo、Chaojie Mao、Xiaotang Gai、Xi Chen 等 18 人

与你有关的部分

将交错多模态生成分解为文本规划与视觉一致性建模,并利用文本代理数据训练规划器、参考引导图像数据训练可视化器,为视觉生成中的长程跨模态上下文建模提供了可复用的解耦训练思路。

原文依据 we decompose interleaved generation into textual planning and visual consistency modeling
可迁移到 文本到图像生成多模态交错生成视觉一致性建模

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

近期统一模型在理解与生成两方面均取得了前所未有的进展。然而,尽管大多数模型接受多模态输入,它们通常仅产生单模态输出。产生交错内容的这一挑战主要源于训练数据稀缺以及长程跨模态上下文建模的困难。为解决该问题,我们将交错生成分解为文本规划与视觉一致性建模,并引入一个由规划器和可视化器组成的框架。规划器为视觉内容生成密集的文本描述,而可视化器则据此合成图像。在此指导下,我们构建了大规模文本代理交错数据(其中视觉内容以文本表示)以训练规划器,并整理了参考引导的图像数据以训练可视化器。这些设计催生了 Wan-Weaver,它展现出具有长程文本连贯性和视觉一致性的 emergent 交错生成能力。同时,将多样化的理解与生成数据整合到规划器训练中,使 Wan-Weaver 能够实现稳健的任务推理与生成能力。为评估模型的交错生成能力,我们进一步构建了一个涵盖多个维度广泛用例的基准。大量实验表明,即使无法获取任何真实交错数据,Wan-Weaver 仍优于现有方法。

英文原文摘要 1501 字符

Recent unified models have made unprecedented progress in both understanding and generation. However, while most of them accept multi-modal inputs, they typically produce only single-modality outputs. This challenge of producing interleaved content is mainly due to training data scarcity and the difficulty of modeling long-range cross-modal context. To address this issue, we decompose interleaved generation into textual planning and visual consistency modeling, and introduce a framework consisting of a planner and a visualizer. The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly. Under this guidance, we construct large-scale textual-proxy interleaved data (where visual content is represented in text) to train the planner, and curate reference-guided image data to train the visualizer. These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency. Meanwhile, the integration of diverse understanding and generation data into planner training enables Wan-Weaver to achieve robust task reasoning and generation proficiency. To assess the model's capability in interleaved generation, we further construct a benchmark that spans a wide range of use cases across multiple dimensions. Extensive experiments demonstrate that, even without access to any real interleaved data, Wan-Weaver achieves superior performance over existing methods.

标签

会议 CVPR2026
页码 36267-36278
Bibkey
来源 CVF Open Access