数据集与评测 · CVPR2026 ·2026.06.03

WEAVE:释放并基准化上下文交错理解与生成

WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation

Wei Chow、Jiachun Pan、Yongyuan Liang、Mingze Zhou、Xue Song、Liyu Jia 等 13 人

与你有关的部分

提供了首个面向多轮上下文交错理解与生成的大规模数据集与人工标注基准,并给出结合参考图像与原始图文指令对的混合 VLM 评判协议,可直接用于训练和评测统一多模态模型的视觉记忆与上下文推理能力。

原文依据 WEAVE-Bench is a human-annotated benchmark of 100 items with 480 images, equipped with a hybrid VLM judge evaluation framework
可迁移到 多模态理解与生成图像编辑视觉问答多轮对话生成

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

近期统一多模态模型(UMM)在视觉理解与生成方面取得了显著进展。然而,现有数据集与基准主要关注单轮交互,忽视了真实世界图像创作与编辑中多轮、依赖上下文的特性。为弥补这一空白,我们提出 WEAVE,首个面向上下文交错跨模态理解与生成的综合套件,包含两个互补组成部分。WEAVE-100k 是一个大规模数据集,包含 10 万个交错样本,覆盖超过 37 万轮对话和 50 万张图像,涵盖需要对先前上下文进行推理的理解、编辑与生成任务。WEAVE-Bench 是一个人工标注的基准,包含 100 个条目与 480 张图像,配备混合 VLM 评判评估框架,该框架联合利用参考图像与原始图像-指令对,跨多个领域评估多轮生成、视觉记忆与世界知识推理。实验表明,在 WEAVE-100k 上训练可显著提升视觉理解、图像编辑以及理解-生成协同能力,并进一步促使统一多模态模型中涌现出视觉记忆能力。在 WEAVE-Bench 上的广泛评估揭示了当前方法在多轮、上下文感知的图像生成与编辑方面仍存在持续局限。我们希望 WEAVE 为多模态社区推进上下文交错理解与生成提供一种视角与基础。

英文原文摘要 1563 字符

Recent unified multimodal models (UMMs) have achieved remarkable progress in visual comprehension and generation. However, existing datasets and benchmarks focus predominantly on single-turn interactions, overlooking the multi-turn, context-dependent nature of real-world image creation and editing. To bridge this gap, we introduce W E A V E, the first comprehensive suite for in-context interleaved cross-modality comprehension and generation, comprising two complementary components. W E A V E-100k is a large-scale dataset of 100,000 interleaved samples spanning over 370,000 dialogue turns and 500,000 images, encompassing comprehension, editing, and generation tasks that demand reasoning over prior context. W E A V E-Bench is a human-annotated benchmark of 100 items with 480 images, equipped with a hybrid VLM judge evaluation framework that jointly leverages reference images and original-image-instruction pairs to assess multi-turn generation, visual memory, and world-knowledge reasoning across diverse domains. Experiments show that training on W E A V E-100k substantially improves vision comprehension, image editing, and comprehension-generation collaboration, while further enabling the emergence of visual-memory capabilities in UMMs. Extensive evaluations on W E A V E-Bench reveal persistent limitations of current approaches in multi-turn, context-aware image generation and editing. We hope W E A V E provides both a perspective and a foundation for advancing in-context interleaved comprehension and generation in the multimodal community.

标签

会议 CVPR2026
页码 15343-15353
Bibkey
来源 CVF Open Access