TAS-LoRA：基于混合低秩适配专家的Transformer架构搜索
TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Experts
将低秩适配（LoRA）作为子网专属参数引入Transformer架构搜索，配合混合专家路由，可缓解超网特征坍缩问题；该思路可迁移到视觉Transformer的轻量化搜索与参数高效微调场景。
原文依据 introduces parameter-efficient low-rank adaptation (LoRA) to enable subnet-specific feature learning, while maintaining computational efficiency
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论Transformer架构搜索（TAS）能够自动发现最优的视觉Transformer（ViT）架构，减少人工设计ViT的工作量。然而，现有TAS方法存在特征坍缩问题，即超网中的子网无法学习到子网特有的特征，这主要归因于超网中的共享权重，从而限制了单个子网的性能。为解决这一问题，我们提出TAS-LoRA，一种新颖的方法，引入参数高效的低秩适配（LoRA）来实现子网特有的特征学习，同时保持计算效率。TAS-LoRA结合了混合低秩适配专家（MoLE）策略，其中轻量级路由器根据子网架构动态分配LoRA专家，并引入分组路由器初始化技术，以在训练早期鼓励专家之间进行多样化的特征学习。在ImageNet以及包括CIFAR-10/100、Flowers、CARS和INAT-19在内的多个迁移学习基准上的大量实验表明，TAS-LoRA有效缓解了特征坍缩，显著提升了优于最优TAS方法的性能。
英文原文摘要 1153 字符
Transformer architecture search (TAS) discovers optimal vision transformer (ViT) architectures automatically, reducing human effort to manually design ViTs. However, existing TAS methods suffer from the feature collapse problem, where subnets within a supernet fail to learn subnet-specific features, mainly due to the shared weights in a supernet, limiting the performance of individual subnets. To address this, we propose TAS-LoRA, a novel method that introduces parameter-efficient low-rank adaptation (LoRA) to enable subnet-specific feature learning, while maintaining computational efficiency. TAS-LoRA incorporates a Mixture-of-LoRA-Experts (MoLE) strategy, where a lightweight router dynamically assigns LoRA experts based on subnet architectures, and introduces a group-wise router initialization technique to encourage diverse feature learning across experts early in training. Extensive experiments on ImageNet and several transfer learning benchmarks, including CIFAR-10/100, Flowers, CARS, and INAT-19, demonstrate that TAS-LoRA mitigates feature collapse effectively, improving performance over state-of-the-art TAS methods significantly.