基于半监督率失真的多模态表征学习用于广义类别发现
Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery
把率失真（rate reduction）目标从单模态扩展到多模态半监督场景，用模态内对齐约束表征的紧凑性与结构性，可作为开放集识别、新类发现类视觉任务中替代纯对比学习的表征学习目标。
原文依据 we propose a novel and effective multi-modal representation learning approach for GCD via Semi-Supervised Rate Reduction
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论广义类别发现（GCD）旨在同时识别已知类别与未知类别，且仅对已知类别给出部分标签，构成一个具有挑战性的开放集识别问题。现有最优的 GCD 方法通常建立在多模态表征学习之上，而这些方法高度重视模态间对齐，而非模态内对齐。本文提出一种新颖且有效的多模态表征学习方法，通过半监督率失真实现 GCD，称为 SSR^2-GCD，借助恰当利用模态内对齐来学习具有理想内在结构性质的跨模态表征。此外，为促进知识迁移，我们利用视觉语言模型所提供的模态间对齐来整合提示词候选。我们在通用与细粒度基准数据集上开展了大量实验，证明了所提方法的优越性能，并验证了利用恰当的模态内对齐的重要性。
英文原文摘要 1045 字符
Generalized Category Discovery (GCD) aims to identify both known and unknown categories, with only partial labels given for the known categories, posing a challenging open-set recognition problem. State-of-the-art approaches for GCD are usually built on multi-modality representation learning, which pays heavily attention upon inter-modality alignment rather than intra-modality alignment. In this paper, we propose a novel and effective multi-modal representation learning approach for GCD via Semi-Supervised Rate Reduction, called SSR^2-GCD, to learn cross-modality representations with desired underlying structure properties via properly harnessing intra-modality alignment. Moreover, to boost knowledge transfer, we integrate prompt candidates by leveraging the inter-modal alignment offered by Vision Language Models. We conduct extensive experiments on generic and fine-grained benchmark datasets, demonstrating superior performance of the proposed approach and verifying the importance of harnessing an proper intra-modality alignment.