IsoCLIP：分解 CLIP 投影器以实现高效的模态内对齐
IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
对做视觉检索或复用 CLIP 图像编码器的人，这篇给出了一个免训练的后处理：从 projector 权重里用谱分析取出跨模态对齐的各向同性子空间，丢掉模态特有的各向异性方向，就能直接提升 image-to-image 检索并降低延迟，可作为一个即插即用的特征变换。
原文依据 this aligned subspace can be directly obtained from the projector weights and that removing the anisotropic directions improves intra-modal alignment
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论像 CLIP 这样的视觉语言模型被广泛用于同时涉及视觉与文本两种模态的跨模态任务。然而，当把单个模态的编码器用于图像到图像检索这类本质上是模态内的任务时，其性能会因模态内错位而下降。本文研究 CLIP 中的模态内错位，重点关注将投影前的图像与文本嵌入映射到共享嵌入空间的投影器所起的作用。通过分析作用于投影特征的余弦相似度的形式及其与对比式 CLIP 损失的相互作用，我们表明其中存在一个跨模态算子，负责在训练中对齐两种模态；还存在第二个模态内算子，它只施加模态内的归一化，对促进模态内对齐毫无作用。通过对跨模态算子进行谱分析，我们识别出一个两种模态对齐良好的近似各向同性子空间，以及各自模态特有的各向异性方向。我们证明该对齐子空间可以直接从投影器权重中获得，并且去除各向异性方向能够改善模态内对齐。在模态内检索与分类基准上的实验表明，我们的免训练方法降低了模态内错位，大幅降低延迟，并在多个预训练 CLIP 类模型上优于现有方法。代码已公开于：https://github.com/simomagi/IsoCLIP。
英文原文摘要 1550 字符
Vision-Language Models like CLIP are extensively used for inter-modal tasks which involve both visual and text modalities. However, when the individual modality encoders are applied to inherently intra-modal tasks like image-to-image retrieval, their performance suffers from the intra-modal misalignment. In this paper we study intra-modal misalignment in CLIP with a focus on the role of the projectors that map pre-projection image and text embeddings into the shared embedding space. By analyzing the form of the cosine similarity applied to projected features, and its interaction with the contrastive CLIP loss, we show that there is an inter-modal operator responsible for aligning the two modalities during training, and a second, intra-modal operator that only enforces intra-modal normalization but does nothing to promote intra-modal alignment. Via spectral analysis of the inter-modal operator, we identify an approximately isotropic subspace in which the two modalities are well-aligned, as well as anisotropic directions specific to each modality. We demonstrate that this aligned subspace can be directly obtained from the projector weights and that removing the anisotropic directions improves intra-modal alignment. Our experiments on intra-modal retrieval and classification benchmarks show that our training-free method reduces intra-modal misalignment, greatly lowers latency, and outperforms existing approaches across multiple pre-trained CLIP-like models. The code is publicly available at: https://github.com/simomagi/IsoCLIP.