混合食谱让 DINO 成为杂食性视觉编码器
A Mixed Diet Makes DINO An Omnivorous Vision Encoder
把 DINOv2 这类单模态基础模型通过「跨模态对齐 + 冻结教师蒸馏」扩展成模态无关编码器，可直接复用于 RGB-D、分割图、深度图等多模态输入的统一特征提取与跨模态检索。
原文依据 a distillation objective that anchors the learned representations to a fully frozen teacher such as DINOv2
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论像 DINOv2 这样的预训练视觉编码器在单模态任务上表现优异。然而，我们观察到它们的特征在不同模态之间对齐很差。例如，同一场景的 RGB 图像与其对应深度图之间的特征嵌入余弦相似度，几乎与两张随机、无关图像的相似度相同。为解决这一问题，我们提出杂食性视觉编码器（Omnivorous Vision Encoder），一个学习模态无关特征空间的新框架。我们以双重目标训练该编码器：第一，最大化同一场景不同模态之间的特征对齐；第二，一个蒸馏目标，将学习到的表征锚定到一个完全冻结的教师模型（如 DINOv2）上。由此得到的学生编码器对给定场景无论输入何种模态（RGB、深度、分割图等）都产生一致、统一的嵌入，从而变得「杂食」。该方法在保留原始基础模型判别性语义的同时，实现了鲁棒的跨模态理解。
英文原文摘要 1078 字符
Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their features are poorly aligned across different modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a novel framework that learns a modality-agnostic feature space. We train the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to a fully frozen teacher such as DINOv2. The resulting student encoder becomes "omnivorous" by producing a consistent, unified embedding for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model.