TIPSv2：以增强的图像块-文本对齐推进视觉-语言预训练
TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment
论文提出的 iBOT++ 让未掩码 token 也直接参与损失，并证明图像块级蒸馏可显著增强稠密图像块-文本对齐（学生甚至超过教师），为需要稠密视觉-语言对齐的下游任务（分割、深度预测、开放词表定位）提供了可复用的预训练配方。
原文依据 a patch-level distillation procedure significantly boosts dense patch-text alignment
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉-语言预训练的最新进展已显著改善了许多下游计算机视觉应用，例如分类、检索、分割和深度预测。然而，这些模型仍然难以具备的一项基础能力，是将稠密的图像块表示与对应概念的文本嵌入对齐。在本工作中，我们研究了这一关键问题，并提出新技术以增强基础视觉-语言模型的这一能力。首先，我们揭示出图像块级蒸馏过程能显著提升稠密的图像块-文本对齐——令人惊讶的是，蒸馏后学生模型的图像块-文本对齐能力大幅超过了教师模型。这一观察启发我们考虑对预训练配方进行修改，由此提出 iBOT++，即对常用的 iBOT 掩码图像目标的升级版本，其中未掩码的 token 也直接参与损失计算。这极大地增强了预训练模型的图像块-文本对齐能力。此外，为提升视觉-语言预训练的效率与效果，我们修改了学习配方中的指数滑动平均设置，并引入一种字幕采样策略，以从不同粒度的合成字幕中获益。结合这些组件，我们开发出 TIPSv2，一个适用于广泛下游应用的新系列图文编码器模型。通过在 9 项任务和 20 个数据集上的全面实验，我们展示了强劲的性能，总体与近期的视觉编码器模型持平或更优。代码与模型已通过项目页面 https://gdm-tipsv2.github.io/ 发布。
英文原文摘要 1683 字符
Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation and depth prediction. However, a fundamental capability that these models still struggle with is aligning dense patch representations with text embeddings of corresponding concepts. In this work, we investigate this critical issue and propose novel techniques to enhance this capability in foundational vision-language models. First, we reveal that a patch-level distillation procedure significantly boosts dense patch-text alignment -- surprisingly, the patch-text alignment of the distilled student model strongly surpasses that of the teacher model. This observation inspires us to consider modifications to pretraining recipes, leading us to propose iBOT++, an upgrade to the commonly-used iBOT masked image objective, where unmasked tokens also contribute directly to the loss. This dramatically enhances patch-text alignment of pretrained models. Additionally, to improve vision-language pretraining efficiency and effectiveness, we modify the exponential moving average setup in the learning recipe, and introduce a caption sampling strategy to benefit from synthetic captions at different granularities. Combining these components, we develop TIPSv2, a new family of image-text encoder models suitable for a wide range of downstream applications. Through comprehensive experiments on 9 tasks and 20 datasets, we demonstrate strong performance, generally on par with or better than recent vision encoder models. Code and models are released via our project page at https://gdm-tipsv2.github.io/.