SigLino：面向凝聚式视觉基础模型的高效多教师蒸馏
SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Models
把 SigLIP2 与 DINOv3 两个教师的知识蒸馏进 Dense/MoE 学生，并给出非对称关系知识蒸馏损失、词元均衡批处理与层次聚类数据采样三条可复用配方，可直接用于构建早期融合 Grounding-VLM 的视觉主干。
原文依据 token-balanced batching that packs varying-resolution images into sequences with uniform token budgets
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论通过多教师蒸馏训练的视觉基础模型为通向统一视觉表征提供了有前景的路径，但此类方法的学习动态与数据效率仍未被充分探索。本文系统研究了面向视觉基础模型的多教师蒸馏，并识别出若干能够在更低计算成本下完成训练的关键因素。我们提出 SigLino，一个高效的凝聚式视觉基础模型系列，同时从 SigLIP2 与 DINOv3 蒸馏知识到 Dense 与混合专家（Mixture-of-Experts）学生模型中。我们表明：(1) 我们提出的非对称关系知识蒸馏损失在实现有效知识迁移的同时保留了各教师的几何性质；(2) 词元均衡批处理将不同分辨率的图像打包为具有统一词元预算的序列，在不损失性能的前提下稳定了跨分辨率的表征学习；(3) 对训练数据进行层次聚类与采样——这一做法通常用于自监督学习——相比随机采样能显著提升多教师蒸馏的样本效率；(4) 所得表征可有效迁移到早期融合的 Grounding-VLM，性能优于从零训练的模型。综合上述发现，我们构建了 OpenLVD200M，一个包含 2 亿张图像的语料库，实验表明其在多教师蒸馏上具有更优的效率。以混合专家架构实例化的 SigLino-MoE 初始化了一个早期融合的 Grounding-VLM，用以替代传统的 ViT-LLM 堆叠结构，相比从零训练的模型展现出更优的性能。我们开源了 OpenLVD200M 与蒸馏得到的检查点。
英文原文摘要 1678 字符
Vision foundation models trained via multi-teacher distillation offer a promising path toward unified visual representations, yet the learning dynamics and data efficiency of such approaches remain underexplored. In this paper, we systematically study multi-teacher distillation for vision foundation models and identify key factors that enable training at lower computational cost. We introduce SigLino, an efficient family of agglomerative vision foundation models that distill knowledge from SigLIP2 and DINOv3 simultaneously into Dense and Mixture-of-Experts students. We show that (1) our Asymmetric Relation-Knowledge Distillation loss preserves the geometric properties of each teacher while enabling effective knowledge transfer, (2) token-balanced batching that packs varying-resolution images into sequences with uniform token budgets stabilizes representation learning across resolutions without sacrificing performance, (3) hierarchical clustering and sampling of training data--typically reserved for self-supervised learning--substantially improves sample efficiency over random sampling for multi-teacher distillation, and (4) the resulting representations transfer effectively to early-fusion Grounding-VLMs, outperforming models trained from scratch. By combining these findings, we curate OpenLVD200M, a 200M-image corpus that demonstrates superior efficiency for multi-teacher distillation. Instantiated in a Mixture-of-Experts, our SigLino-MoE initializes an early-fusion Grounding-VLM that replaces the conventional ViT-LLM stack, demonstrating improved performance compared to a model trained from scratch. We release OpenLVD200M and distilled checkpoints.