OrthoFuse：面向扩散模型的正交风格-概念适配器免训练黎曼融合
OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models
对做个性化/风格化扩散生成的人，这提供了一条无需再训练就能把主体适配器与风格适配器合并的路径：利用 GS 正交矩阵所在流形的测地线近似做乘法式适配器融合，并配合谱恢复变换保住融合后的频谱特性，可直接插到现有 OFT/LoRA 式微调流程里。
原文依据 the first training-free method for merging multiplicative orthogonal adapters
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在模型训练领域快速发展的背景下，人们对参数高效微调以及各类利用少量训练数据将模型适配到狭窄任务的技术始终抱有实际的兴趣。然而，仍有一个悬而未决的问题：如何将针对不同任务调优的多个适配器组合成一个能在两个任务上都给出足够好结果的适配器？具体而言，面向生成模型的主体适配器与风格适配器的合并仍未得到解决。本文试图表明，在正交微调（OFT）的情形下，我们可以利用结构化的正交参数化及其几何性质，得到免训练适配器合并的公式。具体来说，我们推导了由近期提出的 Group-and-Shuffle（GS）正交矩阵所构成的流形的结构，并得到了两点之间测地线近似的高效公式。此外，我们提出了一种谱恢复变换，用于恢复合并后适配器的谱性质，从而实现更高质量的融合。我们在主体驱动生成任务上开展了实验，结果表明，我们合并两个 GS 正交矩阵的技术能够将不同适配器的概念特征与风格特征统一起来。据我们所知，这是首个用于合并乘法式正交适配器的免训练方法。代码见链接：https://github.com/ControlGenAI/OrthoFuse。
英文原文摘要 1417 字符
In a rapidly growing field of model training there is a constant practical interest in parameter-efficient fine-tuning and various techniques that use a small amount of training data to adapt the model to a narrow task. However, there is an open question: how to combine several adapters tuned for different tasks into one which is able to yield adequate results on both tasks? Specifically, merging subject and style adapters for generative models remains unresolved. In this paper we seek to show that in the case of orthogonal fine-tuning (OFT), we can use structured orthogonal parametrization and its geometric properties to get the formulas for training-free adapter merging. In particular, we derive the structure of the manifold formed by the recently proposed Group-and-Shuffle (GS) orthogonal matrices, and obtain efficient formulas for the geodesics approximation between two points. Additionally, we propose a spectra restoration transform that restores spectral properties of the merged adapter for higher-quality fusion. We conduct experiments in subject-driven generation tasks showing that our technique to merge two GS orthogonal matrices is capable to unite concept and style features of different adapters. To the best of our knowledge, this is the first training-free method for merging multiplicative orthogonal adapters. Code is available via the link: https://github.com/ControlGenAI/OrthoFuse.