双向模型缩放中知识迁移的统一框架
A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling
把模型权重当作连续信号、用 DWT/IDWT 的分解层级作为缩放因子，可在不重新预训练的情况下把 ViT/DeiT 类视觉骨干在大小之间迁移，为视觉模型扩容或压缩提供免训练的参数初始化手段。
原文依据 treat model weights as continuous signals, where models of different sizes represent distinct discretizations of the transferable knowledge
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论将预训练知识从源模型迁移到不同架构规模的目标模型，是实现灵活高效模型缩放的关键挑战。然而，现有的参数空间方法将小到大（S2L）与大到小（L2S）缩放视为彼此独立、互不兼容的问题，分别侧重于参数合成与参数选择。这种割裂的视角导致工具专门化，阻碍了统一的双向框架的形成。本文提出 BoT（双向知识迁移），这是首个统一 S2L 与 L2S 缩放的规模无关框架。我们的核心洞见是将模型权重视为连续信号，不同规模的模型代表可迁移知识的不同离散化。这种多分辨率视角直接将 S2L 与 L2S 缩放转化为上采样与下采样这两种信号处理操作，从而自然地引入离散小波变换（DWT）及其逆变换（IDWT）。BoT 利用小波的递归特性，以分解层级作为动态缩放因子，以无参数且计算高效的方式弥合不同规模的模型。在 DeiT、BERT 和 GPT 上的大量实验表明，该方法显著节省了预训练浮点运算量（S2L 最高节省 67.1%，L2S 最高节省 52.8%），并在 GLUE 和 SQuAD 等基准上取得最优性能。
英文原文摘要 1395 字符
Transferring pre-trained knowledge from a source model to a target model of a different architectural size is a key challenge for flexible and efficient model scaling. However, current parameter-space methods treat Small-to-Large (S2L) and Large-to-Small (L2S) scaling as separate, incompatible problems, focusing on parameter synthesis and selection, respectively.This fragmented perspective has resulted in specialized tools, hindering a unified, bidirectional framework.In this paper, we propose BoT (Bidirectional knowledge Transfer), the first size-agnostic framework to unify S2L and L2S scaling.Our core insight is to treat model weights as continuous signals, where models of different sizes represent distinct discretizations of the transferable knowledge.This multi-resolution perspective directly casts S2L and L2S scaling as the signal processing operations of upsampling and downsampling, naturally leading to the adoption of the Discrete Wavelet Transform (DWT) and its Inverse (IDWT).BoT leverages the recursive nature of wavelets, using the decomposition level as a dynamic scaling factor to bridge disparate model sizes in a parameter-free and computationally efficient manner. Extensive experiments on DeiT, BERT, and GPT demonstrate significant pre-training FLOPs savings (up to 67.1% for S2L, 52.8% for L2S) and state-of-the-art performance on benchmarks like GLUE and SQuAD.