FINE：分解知识以实现可变尺寸扩散模型的初始化
FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models
把扩散模型每层权重分解为跨层共享的尺寸无关因子 U、V 与层专属的 Σ，从而在只有小模型预训练权重时直接重组初始化任意尺寸的扩散模型，仅需少量数据重训 Σ；对需要在显存受限场景下部署不同规模扩散模型的视觉研究者提供了可复用的权重分解与初始化思路。
原文依据 FINE represents each layer's weights as the product of U_ \star , \Sigma_ \star ^ (l) , and V_ \star ^\top
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论扩散模型的训练计算开销巨大，因此有效的预训练至关重要。然而，由于显存与计算约束各不相同，实际部署往往需要不同尺寸的模型，当相应的预训练版本不可用时便会带来挑战。为解决这一问题，我们提出 FINE，一种新的预训练方法，其所得模型能够将知识灵活分解为称为 learngene 的基本组件，从而直接初始化各种尺寸的模型，无需重复预训练。FINE 并不优化传统的全参数模型，而是将每一层的权重表示为 U_\star、\Sigma_\star^{(l)} 与 V_\star^\top 的乘积，其中 U_\star 与 V_\star 作为跨层共享的、与尺寸无关的 learngene，而 \Sigma_\star^{(l)} 保持层专属。通过联合训练这些组件，FINE 形成了一种可分解、可迁移的知识结构，使得通过灵活重组 learngene 即可高效初始化，只需在有限数据上对 \Sigma_\star^{(l)} 进行轻量重训练。大量实验证明了 FINE 的高效性，在多种资源受限部署下初始化可变尺寸模型时取得了最优性能。此外，由 FINE 初始化的模型能够有效适配多种任务，展现了 learngene 的任务无关通用性。
英文原文摘要 1450 字符
The training of diffusion models is computationally intensive, making effective pre-training essential. However, real-world deployments often demand models of variable sizes due to diverse memory and computational constraints, posing challenges when corresponding pre-trained versions are unavailable.To address this, we propose FINE, a novel pre-training method whose resulting model can flexibly factorize its knowledge into fundamental components, termed learngenes, enabling direct initialization of models of various sizes and eliminating the need for repeated pre-training.Rather than optimizing a conventional full-parameter model, FINE represents each layer's weights as the product of U_ \star , \Sigma_ \star ^ (l) , and V_ \star ^\top, where U_ \star and V_ \star serve as size-agnostic learngenes shared across layers, while \Sigma_ \star ^ (l) remains layer-specific.By jointly training these components, FINE forms a decomposable and transferable knowledge structure that allows efficient initialization through flexible recombination of learngenes, requiring only light retraining of \Sigma_ \star ^ (l) on limited data.Extensive experiments demonstrate the efficiency of FINE, achieving state-of-the-art performance in initializing variable-sized models across diverse resource-constrained deployments. Furthermore, models initialized by FINE effectively adapt to diverse tasks, showcasing the task-agnostic versatility of learngenes.