TaskIT：通过跨任务重要性迁移实现多 LoRA 大语言模型的显存高效微调
TaskIT: Memory-Efficient Fine-Tuning of Multi-LoRA LLMs via Cross-Task Importance Transfer
对需要在显存受限设备上部署多任务视觉-语言模型的场景，TaskIT 的跨任务重要性迁移与基于块的显存预测器可用于在插入 LoRA 模块前估计其重要性，并配合动态规划调度器在显存预算内选择模块位置、数量与秩，可直接迁移到多任务视觉基础模型的端侧微调。
原文依据 TaskIT predicts pre-insertion module importance by transferring from previously tuned tasks and employs a block-based memory predictor
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论端侧 AI 系统越来越多地采用单个基础模型并配备任务特定的低秩适配（LoRA）模块，形成支持多任务的多 LoRA 大语言模型。本文研究如何在显存受限的设备上将此类模型适配到新任务。尽管 LoRA 减少了可训练参数，但微调整套模块仍然显存开销很大。为提升效率，我们采用稀疏更新，在显存预算内只训练一部分 LoRA 模块。然而，现有稀疏更新方法假设所有候选参数均已实例化，无法估计尚不存在的模块的重要性；而此前为串行网络设计的显存模型也无法刻画 Transformer 的分块并行结构。我们提出 TaskIT，一个通过跨任务重要性迁移实现显存高效微调的框架。TaskIT 通过从先前已调优任务迁移来预测插入前模块的重要性，并采用基于块的显存预测器来刻画 Transformer 块的并行与串行依赖关系。随后，动态规划调度器在显存预算内选择模块的位置、数量与秩，以最大化准确率。在单模态与跨模态基准上的实验表明，与 Zero-FT、非 LoRA 以及基于 LoRA 的微调方法相比，TaskIT 取得了更优的准确率-显存权衡。
英文原文摘要 1377 字符
On-device AI systems increasingly adopt a single foundation model equipped with task-specific Low-Rank Adaptation (LoRA) modules, forming a multi-LoRA LLM that supports multiple tasks.We study how to adapt such a model to a new task on memory-constrainted devices.Although LoRA reduces trainable parameters, fine-tuning a full set of modules remains memory-intensive.To improve efficiency, we apply sparse updating, training a subset of LoRA modules within the memory budget.However, existing sparse updating methods assume all candidate parameters are instantiated and cannot estimate the importance of modules that do not yet exist, while prior memory models designed for sequential networks fail to capture the blockwise parallel structure of Transformers.We propose TaskIT, a framework for memory-efficient fine-tuning via cross-task importance transfer. TaskIT predicts pre-insertion module importance by transferring from previously tuned tasks and employs a block-based memory predictor that captures parallel and sequential dependencies of Transformer blocks. A dynamic programming scheduler then selects module locations, numbers, and ranks to maximize accuracy within the memory budget.Experiments on uni-modal and cross-modal benchmarks show that TaskIT achieves superior accuracy-memory tradeoffs compared with Zero-FT, non-LoRA, and LoRA-based fine-tuning methods.