ReFTA：突破张量化参数高效微调中的权重重构瓶颈
ReFTA: Breaking the Weight Reconstruction Bottleneck in Tensorized Parameter-Efficient Fine-Tuning
对需要在 ViT-Large 等大模型上做下游适配的视觉研究者，ReFTA 用张量代数消去训练时的权重重构开销，并只微调主张量分量，可在比 LoRA 少 96% 可训练参数的情况下取得更高精度，是替换 LoRA/PiSSA 的即插即用微调方案。
原文依据 eliminating weight tensor reconstruction via tensor algebraic properties
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于张量的方法因能够减少可训练参数，并且在参数高效微调（如 LoRA 与 PiSSA）中相较基于矩阵的方法具有优势（尤其是在捕捉层间相关性方面），正受到越来越多的关注。然而，直接应用张量方法需要在训练过程中反复重构模型权重，这带来了可观的开销，并成为部署的关键瓶颈。为解决这一局限，我们提出无重构的张量化适配方法（Reconstruction-Free Tensor-Based Adaptation，ReFTA），它具有四点优势：(1) 借助张量代数性质消除权重张量重构；(2) 通过仅微调主张量分量来降低初始化量化误差；(3) 提供根植于张量积代数的严格泛化保证；(4) 实现由单一张量秩配置控制的统一设计。在图像分类（IC）与自然语言理解（NLU）任务上的大量实验表明，ReFTA 在所有被评估方法中取得了最佳的精度—效率权衡。在大多数情况下，ReFTA 以最少的可训练参数取得最高的平均精度。在 ViT-Large 模型的图像分类任务上，ReFTA 的平均精度比 LoRA 高出 5.6%，同时相较 LoRA 最多减少 96% 的可训练参数。在 RoBERTa-Large 的 NLU 任务上，ReFTA 相较大多数现有方法将平均精度提升约 5%，而所用参数比 LoRA（r=1）少 86.4%，比 PiSSA 少 97.5%。代码已发布于 https://github.com/jzheng20/ReFTA。
英文原文摘要 1653 字符
Tensor-based methods have attracted growing interest due to their ability to reduce trainable parameters and offer advantages over matrix-based approaches in parameter-efficient fine-tuning (e.g., LoRA and PiSSA), particularly in capturing inter-layer correlations. However, directly applying tensor methods requires repeated reconstruction of model weights during training, introducing substantial overhead and becoming a key bottleneck for deployment. To address this limitation, we propose Reconstruction-Free Tensor-Based Adaptation (ReFTA), which offers four advantages: (1) eliminating weight tensor reconstruction via tensor algebraic properties; (2) reducing initialization quantization error by fine-tuning only the principal tensor components; (3) providing a rigorous generalization guarantee rooted in tensor product algebra; and (4) enabling a unified design controlled by a single tensor rank configuration. Extensive experiments on both image classification (IC) and natural language understanding (NLU) tasks demonstrate that ReFTA achieves the best accuracy-efficiency trade-off among all evaluated methods. Across most cases, ReFTA attains the highest average accuracy with the fewest trainable parameters. On IC tasks with the ViT-Large model, ReFTA surpasses LoRA by 5.6% in average accuracy, while reducing the number of trainable parameters by up to 96% compared to LoRA. On NLU tasks with RoBERTa-Large, ReFTA improves the average accuracy by approximately 5% over most existing methods while using only 86.4% fewer parameters than LoRA (r=1) and 97.5% fewer than PiSSA. The code is available at https://github.com/jzheng20/ReFTA.