DPL：解耦原型学习以增强视觉-语言 Transformer 对缺失模态的鲁棒性
DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalities
对做多模态视觉-语言模型的人，DPL 提供了一种可插拔的预测头设计：按当前缺失模态情形选择类原型并分解为图像/文本分量，可直接替换现有提示词框架后的全连接检测头，用于提升图像缺失时的推理鲁棒性。
原文依据 We introduce Decoupled Prototype Learning (DPL), a new prediction head architecture that explicitly adjusts its decision process to the observed input modalities.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当某一输入模态（例如图像）缺失时，视觉-语言 Transformer 的性能会急剧下降，因为模型被迫使用不完整的信息进行预测。现有的缺失感知提示词方法有助于缓解这种性能退化，但它们仍依赖传统的预测头（例如全连接层），无论哪个模态存在或缺失，都以相同的方式计算类别分数。我们提出解耦原型学习（DPL），这是一种新的预测头架构，能够根据观测到的输入模态显式调整其决策过程。对于每个类别，DPL 选择一组特定于当前缺失模态情形（图像缺失、文本缺失或混合缺失）的原型。随后，每个原型被分解为图像特定分量和文本特定分量，使该检测头能够依据实际存在的信息做出决策。这种自适应设计使 DPL 能够更有效地处理模态缺失的输入，同时与现有的基于提示词的框架完全兼容。在 MM-IMDb、UPMC Food-101 和 Hateful Memes 上的大量实验表明，DPL 在所有广泛使用的多模态图像-文本数据集以及各种缺失情形下均优于最优方法。
英文原文摘要 1281 字符
The performance of Vision-Language Transformers drops sharply when an input modality (e.g., image) is missing, because the model is forced to make predictions using incomplete information. Existing missing-aware prompt methods help reduce this degradation, but they still rely on conventional prediction heads (e.g., a Fully-Connected layer) that compute class scores in the same way regardless of which modality is present or absent. We introduce Decoupled Prototype Learning (DPL), a new prediction head architecture that explicitly adjusts its decision process to the observed input modalities. For each class, DPL selects a set of prototypes specific to the current missing-modality cases (image-missing, text-missing, or mixed-missing). Each prototype is then decomposed into image-specific and text-specific components, enabling the head to make decisions that depend on the information actually present. This adaptive design allows DPL to handle inputs with missing modalities more effectively while remaining fully compatible with existing prompt-based frameworks. Extensive experiments on MM-IMDb, UPMC Food-101, and Hateful Memes demonstrate that DPL outperforms state-of-the-art approaches across all widely used multimodal image-text datasets and various missing cases.