基于掩码双路径蒸馏的渐隐侧网络内存高效迁移学习
Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation
对做视觉大模型适配的人，这篇给出了一个可直接复用的思路：微调时让冻结主干与可学习侧网络互相蒸馏，推理时直接丢掉侧网络，从而在不增加推理开销的前提下获得侧网络的适配收益；其面向多层编码器的特征蒸馏设计也可迁移到 ViT 类主干的 PEFT 方案中。
原文依据 mutually distilling the frozen backbones and learnable side networks in fine-tuning, and discard the side network during inference
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论内存高效的迁移学习（METL）方法近期在将预训练模型适配到下游任务方面取得了有前景的性能。这类方法避免在大型主干网络中执行梯度反向传播，从而显著减少了可训练参数数量以及微调时的高内存消耗。然而，由于它们通常采用轻量且可学习的侧网络，这些方法在推理时不可避免地引入额外的内存与时间开销，这与高效迁移学习的最终目标相矛盾。为解决上述问题，我们提出了一种名为掩码双路径蒸馏（MDPD）的新方法，在借助渐隐侧网络保持微调阶段的参数与内存效率的同时加速推理。具体而言，MDPD 构建了一个框架：在微调过程中通过冻结主干与可学习侧网络之间的相互蒸馏来提升性能，并在推理阶段丢弃侧网络而不牺牲精度。此外，我们针对具有多层的编码器结构设计了一种新的基于特征的知识蒸馏方法。在视觉、纯语言以及视觉-语言任务的不同主干网络上开展的大量实验表明，我们的方法在保持参数与内存消耗相当的情况下，不仅将推理加速至少 25.2%，而且相比当前最先进的方法显著提升了精度。源代码见 https://github.com/Zhang-VKk/MDPD。
英文原文摘要 1505 字符
Memory-efficient transfer learning (METL) approaches have recently achieved promising performance in adapting pre-trained models to downstream tasks. They avoid applying gradient backpropagation in large backbones, thus significantly reducing the number of trainable parameters and high memory consumption during fine-tuning. However, since they typically employ a lightweight and learnable side network, these methods inevitably introduce additional memory and time overhead during inference, which contradicts the ultimate goal of efficient transfer learning. To address the above issue, we propose a novel approach dubbed Masked Dual Path Distillation (MDPD) to accelerate inference while retaining parameter and memory efficiency in fine-tuning with fading side networks. Specifically, MDPD develops a framework that enhances the performance by mutually distilling the frozen backbones and learnable side networks in fine-tuning, and discard the side network during inference without sacrificing accuracy. Moreover, we design a novel feature-based knowledge distillation method for the encoder structure with multiple layers. Extensive experiments on distinct backbones across vision/language-only and vision-and-language tasks demonstrate that our method not only accelerates inference by at least 25.2% while keeping parameter and memory consumption comparable, but also remarkably promotes the accuracy compared to SOTA approaches. The source code is available at https://github.com/Zhang-VKk/MDPD.