以对比信号引导基于扩散的重建以获得均衡的视觉表征
Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation
把对比信号注入以 CLIP 视觉 token 为条件的扩散重建过程，并用重建图像而非原图产生对比监督，从而在统一目标下同时提升判别性与细节感知能力；这一「重建+对比」联合优化思路可直接迁移到任何需要增强视觉编码器表征的 VLM 流程中。
原文依据 we integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论对比语言-图像预训练（CLIP）中视觉编码器的理解能力有限，已成为下游性能的关键瓶颈。该能力既包括反映类别可分性的判别能力（D-Ability），也包括关注细粒度视觉线索的细节感知能力（P-Ability）。近期的一些方案利用扩散模型，以 CLIP 视觉 token 为条件进行图像重建来增强表征。我们认为这类范式可能损害判别能力，因而无法有效解决 CLIP 的表征局限。为此，我们将对比信号引入基于扩散的重建，以追求更全面的视觉表征。我们首先给出一个直接的设计，即在输入图像上以对比学习增强扩散过程。然而实验结果表明，这种朴素组合会遭遇梯度冲突，性能欠佳。为平衡优化过程，我们提出扩散对比重建（DCR），统一了学习目标。其核心思想是：注入扩散过程的对比信号来自每张重建图像，而非原始输入。我们的理论分析表明，DCR 损失能够联合优化判别能力与细节感知能力。在多种基准和多模态大语言模型上的大量实验验证了本方法的有效性。代码见 https://github.com/boyuh/DCR。
英文原文摘要 1541 字符
The limited understanding capacity of the visual encoder in Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance. This capacity includes both Discriminative Ability (D-Ability), which reflects class separability, and Detail Perceptual Ability (P-Ability), which focuses on fine-grained visual cues. Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens. We argue that such paradigms may compromise D-Ability and therefore fail to effectively address CLIP's representation limitations. To address this, we integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations. We begin with a straightforward design that augments the diffusion process with contrastive learning on input images. However, empirical results show that the naive combination suffers from gradient conflict and yields suboptimal performance. To balance the optimization, we introduce the Diffusion Contrastive Reconstruction (DCR), which unifies the learning objective. The key idea is to inject contrastive signals derived from each reconstructed image, rather than from the original input, into the diffusion process. Our theoretical analysis shows that the DCR loss can jointly optimize D-Ability and P-Ability. Extensive experiments across various benchmarks and multi-modal large language models validate the effectiveness of our method. The code is available at https://github.com/boyuh/DCR.