DuoGen：迈向自主交错多模态生成
DuoGen: Towards Autonomous Interleaved Multimodal Generation
用预训练 MLLM 做视觉理解、用视频预训练 DiT 做图像生成，再以两阶段解耦对齐把两者拼成交错图文生成器，为视觉研究者提供了免单模态预训练即可搭建统一生成模型的可行路径，以及交错图文评测基准。
原文依据 leverages the strong visual understanding of a pretrained multimodal LLM and the visual generation capabilities of a diffusion transformer (DiT)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论交错多模态生成使能力超越单模态生成模型，例如分步操作指南、视觉规划以及为推理生成视觉草稿。然而，现有交错生成模型在通用指令下的质量仍受限于训练数据不足与基础模型容量。我们提出 DuoGen，一个系统性地处理数据构建、架构设计与评测的交错生成框架。在数据方面，我们通过整合由精选原始网站改写而来的多模态对话，以及覆盖日常场景的多样化合成样例，构建了一个大规模高质量指令微调数据集。在架构上，DuoGen 利用预训练多模态大语言模型强大的视觉理解能力，以及在海量视频生成上预训练的扩散 Transformer（DiT）的视觉生成能力，从而避免昂贵的单模态预训练，并支持灵活的基础模型选择。两阶段解耦策略先对 MLLM 进行指令微调，再利用精选的交错图文序列使 DiT 与其对齐。在公开基准与新提出的基准上，DuoGen 在文本质量、图像保真度和图像-上下文对齐方面均优于此前的开源模型，并在统一生成模型中于文本到图像和图像编辑任务上取得最优性能。数据与代码发布于 https://research.nvidia.com/labs/dir/duogen/。
英文原文摘要 1468 字符
Interleaved multimodal generation enables capabilities beyond unimodal generation models, such as step-by-step instructional guides, visual planning, and generating visual drafts for reasoning. However, the quality of existing interleaved generation models under general instructions remains limited by insufficient training data and base model capacity. We present DuoGen, an interleaved generation framework that systematically addresses data curation, architecture design, and evaluation. On the data side, we build a large-scale, high-quality instruction-tuning dataset by combining multimodal conversations rewritten from curated raw websites, and diverse synthetic examples covering everyday scenarios. Architecturally, DuoGen leverages the strong visual understanding of a pretrained multimodal LLM and the visual generation capabilities of a diffusion transformer (DiT) pretrained on video generation, avoiding costly unimodal pretraining and enabling flexible base model selection. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences. Across public and newly proposed benchmarks, DuoGen outperforms prior open-source models in text quality, image fidelity, and image-context alignment, and also achieves state-of-the-art performance on text-to-image and image editing among unified generation models. Data and code are released at https://research.nvidia.com/labs/dir/duogen/