超越图模型：基于随机图适配器的可靠 VLM 微调
Beyond Graph Model: Reliable VLM Fine-Tuning via Random Graph Adapter
把类别文本描述建模成顶点随机知识图，用概率消息传播得到上下文感知的分布表示，再经重参数化采样学习文本适配器；做 VLM 下游少样本分类时可直接替换确定性文本适配器，并配合多模型不确定性融合提升鲁棒性。
原文依据 we propose to exploit random graph model into VLM adapter and develop a novel Vertex Random Graph Adapter (VRGAdapter)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于文本适配器的微调方法在将预训练视觉语言模型（VLM）的知识迁移到下游任务方面已展现出显著潜力。现有工作通常采用确定性的文本特征适配器来精炼每个类别的文本表示。然而，由于属性与上下文等固有因素，每个类别的文本描述存在显著的多样性。这种描述多样性提供了丰富的判别性语义知识，可惠及下游视觉学习任务。显然，传统的确定性适配器模型无法充分捕捉这种多样的语义信息。此外，在 VLM 适配器中利用类间关系也是值得追求的目标。为解决这些问题，我们提出将随机图模型引入 VLM 适配器，并开发了一种新的顶点随机图适配器（VRGAdapter）。VRGAdapter 首先借助顶点随机知识图（VRKG）模型，同时建模每个类别固有的多样描述以及不同类别之间的类间关系。随后，它在 VRKG 上执行概率消息传播，为每个类别节点学习上下文感知的分布表示。最后，它采用重参数化采样函数来实现文本适配器学习。值得注意的是，VRGAdapter 提供了一种更通用的适配器方案，将传统的基于图的适配器作为其特例涵盖在内。此外，为使下游任务获得更鲁棒的性能，我们还引入了一种新的不确定性引导多分支融合（UMF）方案，动态集成多个预训练模型以进行集成预测。在多个基准数据集上的大量实验证明了该方法的有效性。
英文原文摘要 1786 字符
Textual adapter-based tuning methods have shown significant potential in transferring knowledge from pre-trained Vision-Language Models (VLMs) to downstream tasks. Existing works generally employ the deterministic textual feature adapter to refine each category textual representation. However, due to inherent factors such as different attributes and contexts, there exists significant diversity in textual descriptions for each category. Such description diversity offers rich discriminative semantic knowledge that can benefit downstream visual learning tasks. Obviously, the traditional deterministic adapter model cannot adequately capture this varied semantic information. Also, it is desirable to exploit the inter-class relationships in VLM adapter. To address these issues, we propose to exploit random graph model into VLM adapter and develop a novel Vertex Random Graph Adapter (VRGAdapter). VRGAdapter first models the inherent diverse descriptions of each category and inter-class relationships of different categories simultaneously by leveraging a Vertex Random Knowledge Graph (VRKG) model. Then, it employs probabilistic message propagation on VRKG to learn context-aware distribution representation for each class node. Finally, it adopts a reparameterized sampling function to achieve textual adapter learning. Note that, VRGAdapter provides a more general adapter solution that encompasses traditional graph-based adapter as a special case. In addition, to enable more robust performance for downstream tasks, we also introduce a new Uncertainty-guided Multi-branch Fusion (UMF) scheme that dynamically integrates multiple pre-trained models for ensemble prediction. Extensive experiments on multiple benchmark datasets demonstrate the effectiveness of our approach.