DINO 吞下 CLIP：面向开放集三维物体检索的超越已知类适配
DINO Eats CLIP: Adapting Beyond Knowns for Open-set 3D Object Retrieval
把冻结 DINO 的多视图特征做分块动态聚合（CAM）替代均值池化，并用 CLIP 的图文对齐空间合成未见类虚拟特征（VFS）来正则化适配，这套「冻结基础模型 + 虚拟类特征正则」思路可直接迁移到其他开放集/开放词表的 3D 或多视图识别任务。
原文依据 we propose Virtual Feature Synthesis (VFS) module to mitigate bias towards known categories explicitly
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉基础模型通过高效适配多视图图像，在开放集三维物体检索（3DOR）上展现出很大潜力。借助语义对齐的潜变量空间，已有工作通常适配 CLIP 编码器来构建基于视图的三维描述子。尽管 CLIP 具有很强的泛化能力，但其缺乏细粒度性促使我们探索更新的自监督编码器——DINO 的潜力。为此，我们提出 DINO Eats CLIP（DEC），一个通过合成未见类数据加以正则化的动态多视图集成新框架。我们首先发现，对冻结 DINO 骨干网络得到的视图特征直接做均值池化即可取得不错性能。然而，进一步适配会导致对已知类平均视图模式的严重过拟合。为应对该问题，我们设计了名为分块与适配模块（CAM）的模块。它将多视图图像切分为块并动态集成局部视图关系，从而得到比标准池化策略更鲁棒的特征。最后，我们提出虚拟特征合成（VFS）模块，以显式缓解对已知类别的偏置。在底层，VFS 利用 CLIP 宽广且预对齐的视觉-语言空间为未见类合成虚拟特征。通过让 DEC 接触这些虚拟特征，我们大幅提升了其开放集判别能力。在标准开放集 3DOR 基准上的大量实验证明了其优越有效性。
英文原文摘要 1472 字符
Vision foundation models have shown great promise for open-set 3D object retrieval (3DOR) through efficient adaptation to multi-view images. Leveraging semantically aligned latent space, previous work typically adapts the CLIP encoder to build view-based 3D descriptors. Despite CLIP's strong generalization ability, its lack of fine-grainedness prompted us to explore the potential of a more recent self-supervised encoder--DINO. To address this, we propose DINO Eats CLIP (DEC), a novel framework for dynamic multi-view integration that is regularized by synthesizing data for unseen classes. We first find that simply mean-pooling over view features from a frozen DINO backbone gives decent performance. Yet, further adaptation causes severe overfitting on average view patterns of known classes. To combat it, we then design a module named Chunking and Adapting Module (CAM). It segments multi-view images into chunks and dynamically integrates local view relations, yielding more robust features than the standard pooling strategy. Finally, we propose Virtual Feature Synthesis (VFS) module to mitigate bias towards known categories explicitly. Under the hood, VFS leverages CLIP's broad, pre-aligned vision-language space to synthesize virtual features for unseen classes. By exposing DEC to these virtual features, we greatly enhance its open-set discrimination capacity. Extensive experiments on standard open-set 3DOR benchmarks demonstrate its superior efficacy.