FedMPT：视觉语言模型的联邦多标签提示微调
FedMPT: Federated Multi-Label Prompt Tuning of Vision-Language Models
在联邦场景下对 VLM 做多标签提示微调时，用 LLM 生成的条件提示与图像 patch 做最优传输来挖掘区域级语义，并用门控融合多条件预测，可缓解客户端异构数据导致的虚假标签共现过拟合；这套「条件提示 + 区域对齐」思路可迁移到其他需要细粒度多标签或开放词表的 VLM 适配任务。
原文依据 we introduce an optimal transport between the condition-enriched prompts and the image patches to uncover multiple region-level semantics
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于视觉语言模型（VLM）的多标签识别（MLR）旨在利用其预训练知识更好地适应复杂识别场景，从而增强模型鲁棒性。然而，对于需要联邦学习的现实去中心化应用而言，将 VLM 适配到每个拥有私有且异构数据的客户端，可能导致模型过拟合虚假的标签相关性，从而在遇到新样本时触发无关类别。为解决该问题，我们以因果模型重新审视面向 MLR 的联邦学习，采用前门调整，并通过放大真实标签共现的中间变量对 MLR 建模过程进行解耦。在我们的分析指导下，提出了 FedMPT，这是首个专为联邦 MLR 设计的方法。FedMPT 的核心思想是利用可泛化的条件来引导联邦 MLR，以抑制错误的标签激活。为此，FedMPT 引入了一条由大语言模型（LLM）驱动的流程，用于解析支配标签依赖关系的潜在条件。此外，我们引入条件增强提示与图像块之间的最优传输，以挖掘多个区域级语义。最后，我们通过精心设计的门控机制，从不同条件生成协同预测。在多个基准数据集上的实验表明，所提方法取得了有竞争力的结果，并在多种设置下优于当前最优方法。
英文原文摘要 1525 字符
Multi-Label Recognition (MLR) based on Vision-Language Models (VLMs) aims to leverage their pre-trained knowledge to better adapt complex recognition scenarios, thereby enhancing model robustness. However, for realistic decentralized applications requiring federated learning, adapting VLMs to each client that possesses private and heterogeneous data can cause the model to overfit spurious label correlations, consequently triggering irrelevant categories when encountering new samples. To tackle this problem, we reconsider the federated learning for MLR with a causal model, in which we adopt a front-door adjustment and decouple the MLR modeling process by intermediate variables that magnify the oracle label co-occurrence. Guided by our analysis, we propose our FedMPT, the first method specifically designed for federated MLR. The core idea of FedMPT is to leverage generalizable conditions to steer federated MLR to mitigate erroneous label activations. To achieve this, FedMPT introduces an Large Language Model (LLM)-driven pipeline to decipher the underlying conditions that govern label dependencies. Furthermore, we introduce an optimal transport between the condition-enriched prompts and the image patches to uncover multiple region-level semantics. Finally, we generate synergistic predictions from different conditions with a crafted gating mechanism. Experiments on multiple benchmark datasets show that our proposed approach achieves competitive results and outperforms SOTA methods under varied settings.