锚点引导的梯度对齐用于不完整多模态学习
Anchor-Guided Gradient Alignment for Incomplete Multimodal Learning
针对视觉语言预训练模型在模态缺失场景下的优化失衡问题，提出以检索重建样本与完整样本共同构成优化锚点来对齐梯度，并设计语义增强适配器生成动态提示；做多模态下游任务（如跨模态检索、VQA）时可直接复用该训练策略提升缺失模态鲁棒性。
原文依据 we propose a novel ANchor-guided Gradient Alignment (ANGA) framework to address this issue
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言预训练（VLP）已在多种多模态学习（MML）任务上取得显著性能。近期，许多工作致力于重建缺失模态，以提升 VLP 模型在不完整 MML 场景下的适应性。然而，这些方法忽视了严重缺失模态条件下的学习失衡问题，即优化过程被重建样本主导，从而削弱了完整样本的表征。本文提出一种新颖的锚点引导梯度对齐（ANGA）框架来解决该问题。具体而言，我们首先检索相似实例以重建缺失模态，从而缓解信息不足。随后引入熵驱动的课程，逐步将可靠的重建样本与完整样本一同纳入，构成优化锚点，引导梯度对齐以缓解学习失衡。此外，我们设计了一个语义增强适配器，利用检索到的实例生成动态提示，进一步增强 VLP 模型的鲁棒性。在广泛使用的数据集上的大量实验表明，ANGA 在多种缺失模态场景下优于最优（SOTA）基线。代码见该仓库。
英文原文摘要 1333 字符
Vision-language pre-training (VLP) has achieved remarkable performance across diverse multimodal learning (MML) tasks. Recently, many efforts have focused on reconstructing missing modalities to improve the adaptability of VLP models in incomplete MML scenarios. However, these approaches overlook the learning imbalance under severe missing-modality conditions, i.e., the optimization process is dominated by reconstructed samples, thereby weakening complete-sample representations. In this paper, we propose a novel ANchor-guided Gradient Alignment (ANGA) framework to address this issue. Specifically, we first retrieve similar instances to reconstruct the missing modalities, thereby alleviating information deficiency. We then introduce an entropy-driven curriculum that progressively incorporates reliable reconstructed samples together with complete ones to form an optimization anchor, which guides gradient alignment to mitigate learning imbalance. Furthermore, we design a semantic-enhanced adapter that leverages the retrieved instances to generate dynamic prompts, further enhancing the robustness of the VLP model. Extensive experiments on widely used datasets demonstrate the superiority of ANGA over state-of-the-art (SOTA) baselines across various missing-modality scenarios. The code is available at this repository.