从Gromov-Wasserstein距离视角重新思考VLM中的模型选择
Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
提出一种仅需推理的指标（Gromov-Wasserstein距离）来预测视觉编码器与LLM对齐后的VLM性能，可用于在完整训练前高效筛选视觉编码器，避免盲目选择大模型或高零样本精度的编码器。
原文依据 we identify that the structural similarity across modalities plays a crucial but previously overlooked role in vision-encoder selection, which we measure using the Gromov-Wasserstein distance as a proxy.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）通过整合视觉编码器，为传统LLM赋予了视觉能力。尽管近期工作探索了视觉编码器与LLM的各种组合，但对于什么使视觉编码器适合VLM对齐，仍缺乏原则性的理解。本文通过在一个精心整理的、来自不同来源的19个预训练视觉编码器集合上进行全面实验，系统地研究了这一问题。我们首先证明，常见做法（如选择最大尺寸或最高零样本准确率的编码器）始终无法识别最优模型。事实上，这些指标与VLM性能仅表现出弱到中等的相关性。这一有趣的发现引出了一个根本问题：视觉编码器的哪些因素在VLM中起作用？通过全面分析，我们发现跨模态的结构相似性在视觉编码器选择中扮演着关键但此前被忽视的角色，我们使用Gromov-Wasserstein距离作为代理来度量它。从理论角度，我们证明跨模态映射的可学习性可以与Gromov-Wasserstein距离可证明地关联起来。在60多次完整VLM训练运行上的实证验证表明，我们提出的仅推理指标显著优于其他模型选择策略，并与最终VLM性能表现出强得多的相关性，从而能够在完整训练之前高效且有效地预测VLM性能。
英文原文摘要 1562 字符
Vision-Language Models (VLMs) have enhanced traditional LLMs with visual capabilities through the integration of vision encoders. While recent works have explored various combinations of vision encoders and LLMs, there still lacks a principled understanding of what makes a vision encoder suitable for VLM alignment. In this paper, we systematically investigate this question via comprehensive experiments on a curated collection of 19 pre-trained vision encoders from diverse sources. We first demonstrate that common practices, such as choosing encoders with the largest size or highest zero-shot accuracy, consistently fail to identify optimal models. In fact, these metrics show only weak to moderate correlation with VLM performance. This intriguing finding begs a fundamental question: What factors of vision-encoders matter in VLM? Through comprehensive analysis, we identify that the structural similarity across modalities plays a crucial but previously overlooked role in vision-encoder selection, which we measure using the Gromov-Wasserstein distance as a proxy. From a theoretical perspective, we show that the learnability of cross-modality mapping can be provably associated with the Gromov-Wasserstein distance. Empirical verification on 60+ full VLM training runs shows that our proposed inference-only metric performs significantly better than alternative model selection strategies and exhibits a much stronger correlation with final VLM performance, thereby enabling efficient and effective prediction of VLM performance before full training.