通过稀疏对齐与单模态记忆库弥合组合零样本学习中的模态间隙
Bridging the Modality Gap in Compositional Zero-Shot Learning via Sparse Alignment and Unimodal Memory Bank
针对 CLIP 图文模态间隙，用稀疏对齐把文本表征直接绑定到语义相关的视觉 patch，并用纯视觉记忆库做分类，这套思路可迁移到任何依赖 CLIP 对齐的零样本/开放词表视觉任务。
原文依据 we conduct sparse alignment that links textual representations directly to their semantically pertinent visual patches
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论组合零样本学习（CZSL）旨在利用从已见组合中学到的基元（属性与物体）知识，识别未见的属性-物体组合。以往方法借助 CLIP 强大的跨模态对齐取得了显著性能，但往往忽视了模态间隙——一种源于训练数据信息不均衡的固有约束。本文提出 SAM，即稀疏对齐与单模态记忆库，以有效弥合 CZSL 中的模态间隙。具体而言，我们进行稀疏对齐，将文本表征直接链接到与其语义相关的视觉 patch。这种直接链接可剪除冗余视觉数据，并抵消图文对中的信息不均衡。随后，以稀疏对齐后的视觉信息为引导，视觉自适应凝聚模块将这些关键线索自适应地融合为统一表征。最后，我们引入一个动态更新的记忆库，存储来自已见与未见组合的样本。该记忆库具有双重作用：通过纯视觉分类绕过模态间隙，同时增强对未见组合的泛化能力。在三个基准上的实验表明，我们的方法在闭世界与开世界设定下相较基于 CLIP 的方法取得了显著提升。
英文原文摘要 1452 字符
Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions with learned primitives (attribute and object) knowledge from seen compositions. While previous approaches gain their notable performance through the powerful cross-modal alignment of CLIP, they often overlook the modality gap, an inherent constraint stemming from information-imbalanced training data. In this work, we propose SAM, a novel \underline \text S parse \underline \text A lignment and Unimoal \underline \text M emory Bank to effectively bridging modality gap for CZSL. Specifically, we conduct sparse alignment that links textual representations directly to their semantically pertinent visual patches. This direct linking serves to prune redundant visual data and counter the information imbalance in image-text pairs. Subsequently, with the sparsely aligned visual information as its guidance, the visual adaptive condensation module adaptively fuses these critical cues into a unified representation. Finally, we introduce a dynamically updated memory bank that stores samples from both seen and unseen compositions. This bank serves a dual purpose: it bypasses the modality gap through visual-only classification and concurrently strengthens generalization to unseen compositions. Experiments on three benchmarks demonstrate that our method gains significant improvements over CLIP-based methods under closed-world and open-world settings.