分解、混合、适配：参数高效神经网络重组与压缩的统一框架
Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression
提出一种统一的权重分解与重组框架，可同时实现参数高效微调（PEFT）和模型压缩（MC），为视觉模型（如ViT）在边缘设备上的部署提供了一种可复用的技术方案。
原文依据 Coefficient-gated weight Recombination by Interpolated Shared basis Projections
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论参数重组（PR）方法旨在高效组合神经网络的权重，涵盖参数高效微调（PEFT）和模型压缩（MC）等任务。大多数方法通常只关注PR的一种应用，这使得将它们组合起来具有挑战性。例如，在部署大型模型时，可能希望压缩模型并快速适应新设置。然而，PEFT方法通常仍可能包含数百万个参数。与原始模型大小相比，这可能很小，但在资源受限的部署（如边缘设备）中可能成为问题，因为它们占据了压缩模型参数的较大比例。为了解决这个问题，我们提出了通过插值共享基投影的系数门控权重重组（Coefficient-gated weight Recombination by Interpolated Shared basis Projections），这是一种通用方法，可以在同一框架内处理多个PR任务，从而实现无缝集成。它通过一种分解过程实现，该过程将预训练权重分解为基矩阵及其分量投影。跨层共享这些基矩阵并调整其大小使我们能够执行MC，而投影权重的小尺寸（在某些实验中少于200个）使该方法支持PEFT。在ViT模型上的实验表明，该方法比先前能够进行双任务应用的方法高出4-5%，同时在PEFT上比最优方法高出1.5%，在PEFT+MC组合上高出近1%。
英文原文摘要 1500 字符
Parameter Recombination (PR) methods aim to efficiently compose the weights of a neural network, and encompasses tasks like Parameter-Efficient FineTuning (PEFT) and Model Compression (MC), among others. Most methods typically focus on one application of PR, which can make composing them challenging. For example, when deploying a large model you may wish to compress the model and also quickly adapt to new settings. However, PEFT methods often can still contain millions of parameters. This may be small compared to the original model size, but can be problematic in resource constrained deployments like edge devices, where they take a larger portion of the compressed model's parameters. To address this, we present Coefficient-gated weight Recombination by Interpolated Shared basis Projections (\method ), a general approach that can address multiple PR tasks within the same framework, which can enable seamless integration. It accomplishes this by using a factorization process that decomposes pretrained weights into basis matrices and their component projections. Sharing these basis matrices across layers and adjusting its size enables us to perform MC, whereas the small size of the projection weights (fewer than 200 in some experiments) enables \method support PEFT. Experiments on ViT models show \method outperforms methods from prior work capable of dual-task applications by 4-5% while also outperforming the state-of-the-art in PEFT by 1.5% and PEFT+MC combinations by almost 1%.