重新评估CLIP中的模态内错位假设
Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP
该论文对CLIP类模型嵌入空间的分析表明，跨模态对齐损失并未导致图像间距离的次优校准，因此视觉研究者无需为图像任务额外引入模态内对齐损失，可直接复用CLIP/SigLIP特征进行检索和少样本分类。
原文依据 Experiments on the commonly studied intra-modal tasks retrieval and few-shot classification confirm that addressing supposed misalignment is unnecessary for best performance.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期研究提出，类CLIP的对比语言-图像训练所产生的嵌入对于纯图像任务而言是次优的。主要理论认为，模态间（语言-图像）对齐损失忽略了模态内（图像-图像）对齐，导致图像间距离校准不佳。本研究对模态内错位假设提出质疑。我们重新审视了其基础理论论证、用于支持该假设的指标以及受影响的性能度量。对于理论论证，我们证明图像嵌入距离并不存在所假设的自由度。对于实证度量，我们的发现表明，语言-图像训练模型（CLIP、SigLIP）与图像-图像训练模型（DINO、SigLIP2）产生相似的结果。这表明观察到的现象并非源于前者特有的错位。在常研究的模态内任务检索和少样本分类上的实验证实，解决所谓的错位对于获得最佳性能并非必要。
英文原文摘要 1074 字符
Recent research suggested that the embeddings produced by CLIP-like contrastive language-image training are suboptimal for image-only tasks. The main theory is that the inter-modal (language-image) alignment loss ignores intra-modal (image-image) alignment, leading to poorly calibrated distances between images. In this study, we question this intra-modal misalignment hypothesis. We reexamine its foundational theoretical argument, the indicators used to support it, and the performance metrics affected. For the theoretical argument, we demonstrate that there are no such supposed degrees of freedom for image embedding distances. For the empirical measures, our findings reveal they yield similar results for language-image trained models (CLIP, SigLIP) and image-image trained models (DINO, SigLIP2). This indicates the observed phenomena do not stem from a misalignment specific to the former. Experiments on the commonly studied intra-modal tasks retrieval and few-shot classification confirm that addressing supposed misalignment is unnecessary for best performance.