DreamOmni2：多模态指令驱动的生成与编辑
DreamOmni2: Multimodal Instruction-based Generation and Editing
提出多模态指令（文本+图像）编辑与生成任务，并设计索引编码与位置编码偏移方案处理多图输入，可复用于需要融合多张参考图的生成/编辑模型；数据合成管线（特征混合+提取模型）为构建此类训练数据提供可借鉴流程。
原文依据 we propose an index encoding and position encoding shift scheme, which helps the model distinguish images and avoid pixel confusion.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期，基于指令的图像编辑和主体驱动生成取得了显著进展，但这两类任务在满足实际用户需求方面仍存在局限。基于指令的编辑仅依赖语言指令，往往无法捕捉具体的编辑细节，因此需要参考图像。同时，主体驱动生成仅限于组合具体物体或人物，忽略了更广泛的抽象概念。为解决这些挑战，我们提出了两个新任务：多模态指令驱动的编辑与生成。这些任务同时支持文本和图像指令，并将范围扩展到具体和抽象概念，极大地增强了实际应用性。我们提出了 DreamOmni2，解决数据创建和模型框架设计两个主要挑战。我们的数据合成管线包含三个步骤：（1）使用特征混合方法为抽象和具体概念创建提取数据；（2）利用编辑和提取模型生成多模态指令驱动的编辑训练数据；（3）进一步应用提取模型为多模态指令驱动的编辑创建训练数据。在框架方面，为处理多图像输入，我们提出了索引编码和位置编码偏移方案，帮助模型区分图像并避免像素混淆。此外，我们引入了视觉语言模型与生成/编辑模型的联合训练，以更好地处理复杂指令。另外，我们为这两个新任务提出了全面的基准以推动其发展。实验表明，DreamOmni2 取得了令人印象深刻的结果。
英文原文摘要 1707 字符
Recent advancements in instruction-based image editing and subject-driven generation have garnered significant attention, yet both tasks still face limitations in meeting practical user needs. Instruction-based editing relies solely on language instructions, which often fail to capture specific editing details, making reference images necessary. Meanwhile, subject-driven generation is limited to combining concrete objects or people, overlooking broader, abstract concepts. To address these challenges, we propose two novel tasks: multimodal instruction-based editing and generation. These tasks support both text and image instructions and extend the scope to include both concrete and abstract concepts, greatly enhancing their practical applications. We introduce DreamOmni2, tackling two primary challenges: data creation and model framework design. Our data synthesis pipeline consists of three steps: (1) using a feature mixing method to create extraction data for both abstract and concrete concepts, (2) generating multimodal instruction-based editing training data using the editing and extraction models, and (3) further applying the extraction model to create training data for multimodal instruction-based editing. For the framework, to handle multi-image input, we propose an index encoding and position encoding shift scheme, which helps the model distinguish images and avoid pixel confusion. Additionally, we introduce joint training with the VLM and our generation/editing model to better process complex instructions. In addition, we have proposed comprehensive benchmarks for these two new tasks to drive their development. Experiments show that DreamOmni2 achieved impressive results.