S2FT：稀疏频谱域中的参数高效微调
S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain
为视觉大模型（ViT、扩散骨干等）的下游适配提供了一种仅需约0.08%可训练参数的PEFT方案：先用粗估的权重变化作为先验，再通过行列重排把权重变化映射到频谱稀疏的潜空间，然后只微调少量谱系数；做迁移学习或低资源微调时，可以直接替换LoRA/傅里叶PEFT的适配模块。
原文依据 we propose to seek an invertible transformation that can transform a latent spatial-domain matrix with sparse spectrum to the weight change
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论参数高效微调（PEFT）是一种通过仅微调少量参数，使大型预训练模型适配下游任务的关键技术。近期基于傅里叶变换的方法通过仅微调少量谱系数，进一步降低了微调参数的规模。其基本假设是权重变化 ΔW 是一个频谱稀疏的空域矩阵。然而，本文观察到权重变化的频谱并不稀疏，而是呈功率均匀分布。这一事实意味着，仅微调少量谱系数不足以准确建模具有均匀频谱的权重变化 ΔW。为解决该问题，我们提出寻找一种可逆变换，将频谱稀疏的潜空域矩阵变换为权重变化，随后在这种稀疏频谱域中仅用少量谱系数执行 PEFT，称为 S2FT。为寻找该变换，我们首先预估计一个粗略的权重变化作为先验。随后，受稀疏频谱通常对应局部平滑空间结构这一现象的启发，我们将该变换视为对预估计权重变化进行行列重排的操作，在平滑空间结构的同时保留神经元的结 构信息。最后，我们提出以简单的最近邻搜索方式求解重排搜索问题，从而获得该可逆变换。大量结果表明，S2FT 仅使用 0.08% 的训练参数即取得优越性能。
英文原文摘要 1616 字符
Parameter Efficient Fine-Tuning (PEFT) is a key technique for adapting a large pretrained model to downstream tasks by fine-tuning only a small number of parameters. Recent methods based on Fourier transforms have further reduced the fine-tuned parameters scale by only fine-tuning a few spectral coefficients. Its basic assumption is that the weight change \Delta W is a spatial-domain matrix with a sparse spectrum. However, in this paper, we observe that the spectrum of weight change is not sparse, but instead distributed like power-uniform. This fact implies that fine-tuning only a few spectral coefficients is insufficient to accurately model the weight change \Delta W with uniform spectrum.To address this issue, we propose to seek an invertible transformation that can transform a latent spatial-domain matrix with sparse spectrum to the weight change, and then perform PEFT on such sparse spectrum domain with few spectral coefficients, called \text S ^2\text FT . To seek such transformation, we first pre-estimate a coarse weight change as a prior. Then, inspired by that sparse spectrum often correspond to locally smooth spatial structures, we regard this transformation as a row and column rearrangement operation on the pre-estimated weight change that smooth spatial structures while keep the structure information of neurons.Finally, we propose to solve the rearrangement search problem in a simple nearest neighbor search manner, thereby obtaining the invertible transformation. Extensive results show our \text S ^2\text FT achieves superior performance by only using 0.08% training parameters.