模态鸿沟是缺陷还是特性？一个鲁棒性视角
Is the Modality Gap a Bug or a Feature? A Robustness Perspective
对做视觉-语言对齐（CLIP 类）的研究者，这篇给出了一个可直接复用的后处理手段：把图像嵌入整体平移到文本嵌入均值方向，即可在不损失干净精度的前提下提升对语义无关扰动的稳定性；同时提供了「对比损失最优解必然产生正交全局鸿沟向量」的理论视角，可用于分析嵌入空间几何与鲁棒性的关系。
原文依据 we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论许多现代多模态模型（如 CLIP）试图寻找一个使两种模态对齐的嵌入空间。令人意外的是，几乎所有现有模型都表现出强烈的模态鸿沟：在共享嵌入空间中，图像的分布与文本的分布彼此明显分离。尽管近期已有一系列论文讨论这一话题，但该鸿沟为何存在、以及在后处理中弥合鸿沟是否会带来下游任务性能的提升，仍不清楚。本文证明，在某些条件下，最小化对比损失会得到这样一种表示：两种模态被一个全局鸿沟向量分隔，且该向量与两种模态的嵌入均正交。我们还证明，在这些条件下，模态鸿沟与鲁棒性单调相关：减小鸿沟不会改变模型的干净精度，但会使模型在输入发生微小、语义上无关紧要的变化时更不容易改变其输出。实验表明，对许多真实世界的视觉语言模型，我们可以通过一个简单的后处理步骤显著提升鲁棒性，即把一种模态向另一种模态的均值移动，且不带来任何干净精度的损失。
英文原文摘要 1244 字符
Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks. In this paper we show that under certain conditions, minimizing the contrastive loss will lead to a representation in which the two modalities are separated by a global gap vector that is orthogonal to the embeddings of both modalities. We also show that under these conditions the modality gap is monotonically related to robustness: decreasing the gap does not change the clean accuracy of the models but makes it less likely that a model will change its output when small, semantically inconsequential changes are made to the input. Our experiments show that for many real-world VLMs we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality, without any loss to clean accuracy.