模型链预训练：重新思考视觉基础模型的训练加速
Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models
CoM-PT 提出在模型家族层面按规模递增顺序进行链式预训练，仅最小模型独立训练，其余模型通过参数空间与特征空间的逆向知识迁移从更小模型高效获得性能，为视觉基础模型的大规模预训练提供了一种可复用的加速范式。
原文依据 only the smallest model undergoes standard individual pre-training, while the other models are efficiently trained through sequential inverse knowledge transfer
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文提出模型链预训练（CoM-PT），一种面向视觉基础模型（VFM）的性能无损训练加速新方法。该方法与现有加速方法的核心动机根本不同：它并非针对单个模型进行优化，而是在模型家族层面加速训练流程，并随模型家族扩展而高效缩放。具体而言，CoM-PT 为模型家族建立按模型规模升序排列的预训练序列，称为模型链。在该链中，仅最小模型经历标准的独立预训练，其余模型则通过从更小前驱模型进行顺序逆向知识迁移，联合复用参数空间与特征空间中的知识，从而高效训练。因此，CoM-PT 使所有模型在显著降低训练成本的同时，获得大多优于标准独立训练的性能，并在涵盖零样本与微调任务的 45 个数据集上得到广泛验证。值得注意的是，其高效缩放特性带来一个显著现象：训练更多模型甚至能带来更高效率。例如，在 CC3M 上预训练时：i）以 ViT-L 为最大模型，逐步在模型链前端添加更小模型可将计算复杂度降低多达 72%；ii）在固定模型规模范围内，随着 VFM 家族扩展到 3、4、7 个模型，CoM-PT 的加速比呈现惊人跃升：从 4.13 倍到 5.68 倍和 7.09 倍。由于 CoM-PT 天然与具体预训练范式无关，我们开源代码以促进其在更多计算密集场景（如大语言模型预训练）中的进一步扩展。
英文原文摘要 1820 字符
In this paper, we present Chain-of-Models Pre-Training (CoM-PT), a novel performance-lossless training acceleration method for vision foundation models (VFMs). This approach fundamentally differs from existing acceleration methods in its core motivation: rather than optimizing each model individually, CoM-PT is designed to accelerate the training pipeline at the model family level, scaling efficiently as the model family expands. Specifically, CoM-PT establishes a pre-training sequence for the model family, arranged in ascending order of model size, called model chain. In this chain, only the smallest model undergoes standard individual pre-training, while the other models are efficiently trained through sequential inverse knowledge transfer from their smaller predecessors by jointly reusing the knowledge in the parameter space and the feature space. As a result, CoM-PT enables all models to achieve performance that is mostly superior to standard individual training while significantly reducing training cost, and this is extensively validated across 45 datasets spanning zero-shot and fine-tuning tasks. Notably, its efficient scaling property yields a remarkable phenomenon: training more models even results in higher efficiency. For instance, when pre-training on CC3M: i) given ViT-L as the largest model, progressively prepending smaller models to the model chain reduces computational complexity by up to 72%; ii) within a fixed model size range, as the VFM family scales across 3, 4, and 7 models, the acceleration ratio of CoM-PT exhibits a striking leap: from 4.13x to 5.68x and 7.09x. Since CoM-PT is naturally agnostic to specific pre-training paradigms, we open-source the code to spur further extensions in more computationally intensive scenarios, such as large language model pre-training.