EditMGT:释放掩码生成Transformer在图像编辑中的潜力
EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editing
用 MGT 的交叉注意力图做编辑区域定位、并在低注意力区限制词元翻转,为视觉研究者提供了一条替代扩散全局去噪的局部保真编辑路径,可直接迁移到 inpainting 与局部风格迁移。
原文依据 MGT's cross-attention maps provide informative localization signals for localizing edit-relevant regions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期扩散模型(DMs)的进展在图像编辑任务中取得了卓越的视觉质量。然而,扩散模型的全局去噪动力学本质上会将局部编辑目标与全图上下文混为一谈,导致非目标区域出现非预期修改。本文跳出扩散模型,转向掩码生成Transformer(MGTs)作为应对该挑战的替代方案。通过预测多个被掩码的词元而非整体式精修,MGTs 展现出一种局部化解码范式,使其在编辑过程中天然具备显式保留无关区域的能力。基于这一洞见,我们提出了首个基于 MGT 的图像编辑框架 EditMGT。我们首先证明 MGT 的交叉注意力图能够提供有信息量的定位信号,用于定位与编辑相关的区域,并设计了一种多层注意力整合方案来精炼这些图,以实现细粒度且精确的定位。在这些自适应定位结果之上,我们引入了区域保持采样,将词元翻转限制在低注意力区域内以抑制虚假编辑,从而把修改约束在预期目标区域并保持周围非目标区域的完整性。为训练 EditMGT,我们构建了 Crisp-2M,一个覆盖七类多样编辑类别的高分辨率(>1024)数据集。在不引入额外参数的情况下,我们通过注意力注入将预训练的文本到图像 MGT 适配为图像编辑模型。在四个标准基准上的大量实验表明,我们的模型在参数量少于 10 亿的情况下取得了最优的图像相似度性能,同时实现了更快的编辑。此外,它还提供了相当或更优的编辑质量,在风格改变和风格迁移任务上分别提升了 3.6% 和 17.6%。
英文原文摘要 1931 字符
Recent advances in diffusion models (DMs) have achieved exceptional visual quality in image editing tasks. However, the global denoising dynamics of DMs inherently conflate local editing targets with the full-image context, leading to unintended modifications in non-target regions. In this paper, we shift our attention beyond DMs and turn to Masked Generative Transformers (MGTs) as an alternative approach to tackle this challenge. By predicting multiple masked tokens rather than holistic refinement, MGTs exhibit a localized decoding paradigm that endows them with the inherent capacity to explicitly preserve non-relevant regions during the editing process. Building upon this insight, we introduce the first MGT-based image editing framework, termed EditMGT. We first demonstrate that MGT's cross-attention maps provide informative localization signals for localizing edit-relevant regions and devise a multi-layer attention consolidation scheme that refines these maps to achieve fine-grained and precise localization. On top of these adaptive localization results, we introduce region-hold sampling, which restricts token flipping within low-attention areas to suppress spurious edits, thereby confining modifications to the intended target regions and preserving the integrity of surrounding non-target areas. To train EditMGT, we construct Crisp-2M, a high-resolution (>1024) dataset spanning seven diverse editing categories. Without introducing additional parameters, we adapt a pre-trained text-to-image MGT into an image editing model through attention injection. Extensive experiments across four standard benchmarks demonstrate that, with fewer than 1B parameters, our model achieves state-of-the-art image similarity performance while enabling faster editing. Moreover, it delivers comparable or superior editing quality, with improvements of 3.6% and 17.6% on style change and style transfer tasks, respectively.