面向扩散生成图像检测的跨模态表征学习
Cross-modal Representation Learning for Diffusion-generated Image Detection
提出跨模态对比学习与互蒸馏，将RGB与邻域像素关系模态的特征对齐，学习伪造感知嵌入空间，可直接用于提升扩散生成图像检测的泛化性。
原文依据 we leverage features from both RGB modality and NPR modality to perform two proposed representation learning methods
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论扩散模型在图像创建与编辑方面惊人的熟练度和前所未有的真实感无疑引发了担忧。已有许多方法被提出用于检测生成图像。通常，它们以RGB图像为输入，并使用ResNet、CLIP视觉编码器等骨干网络提取特征。尽管这些骨干网络能够检测伪造图像，但它们主要设计用于提取高层语义信息，而非天生为伪造图像检测而设计。为此，本文希望通过表征学习来优化专为检测伪造图像而设计的嵌入空间。我们注意到邻域像素关系（NPR）能够捕捉内在的伪造线索，这意味着NPR可能是执行表征学习以学习专为检测伪造图像而设计的嵌入空间的良好输入。因此，我们利用来自RGB模态和NPR模态的特征来执行两种提出的表征学习方法——跨模态对比学习（CMCL）和跨模态互蒸馏（CMMD），以学习伪造感知的嵌入空间。CMCL提升了真实与伪造图像之间特征的判别性，而CMMD同时在两模态间迁移学习到的知识，能够学习类内紧凑特征。CMCL与CMMD协同工作，使每个模态学习到更全面的伪造感知表征以区分真实与伪造图像。在GenImage、DRCT-2M和Co-Spy-Bench数据集上的大量实验表明，我们的方法达到了最优结果。
英文原文摘要 1681 字符
The astonishing proficiency and unprecedented level of realism of diffusion models in creating and manipulating images have undoubtedly drawn concerns.Many methods have been proposed to detect generated images. Typically, they usually take RGB images as input, and use backbones like ResNet, CLIP visual encoder to extract features. Even though these backbones are capable to detect fake images, they are mainly designed to extract the high-level semantic information, rather than inherently designed for fake image detection. To this end, in this paper, we want to optimize the embedding space tailored for detecting fake images via representation learning. We notice that Neighboring Pixel Relationships (NPR) is capable to capture the intrinsic forgery clues, which means that NPR may be a good input to perform representation learning that aims at learning the embedding space tailored for detecting fake images.Therefore, we leverage features from both RGB modality and NPR modality to perform two proposed representation learning methods, Cross-Modal Contrastive Learning (CMCL) and Cross-Modal Mutual Distillation (CMMD), in order to learn the forgery-aware embedding space. The CMCL boosts the discrimination of features between real and fake images, while the CMMD simultaneously transfers the learned knowledge between two modalities, being able to learn compact features within the intra-class. CMCL and CMMD work collaboratively so that each modality learns a more comprehensive forgery-aware representation to distinguish real and fake images.Extensive experiments on GenImage, DRCT-2M, and Co-Spy-Bench datasets show that our method achieves state-of-the-art results.