CRAFT-LoRA:通过秩约束适配与无训练融合实现内容-风格个性化
CRAFT-LoRA: Content-Style Personalization via Rank-Constrained Adaptation and Training-Free Fusion
把多个 LoRA 权重在推理时无训练地组合起来,并通过秩约束微调让内容与风格落在解耦子空间,可直接用于视觉生成中的多概念个性化与风格迁移;其按时间步调整噪声预测的无分类器引导方案也可迁移到其他扩散采样流程以提升稳定性。
原文依据 a training-free, timestep-dependent classifier-free guidance scheme that enhances generation stability by strategically adjusting noise predictions across diffusion steps
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论个性化图像生成需要在基于文本与参考样例合成图像时,有效平衡内容保真度与风格一致性。低秩适配(LoRA)提供了一种高效的个性化途径,并具有通过组合不同概念上的 LoRA 权重来实现精确控制的潜力。然而,现有的组合技术面临长期存在的挑战:内容与风格表示相互纠缠、对控制各元素影响力的引导不足,以及权重融合不稳定而往往需要额外训练。我们通过 CRAFT-LoRA 及其互补组件来解决这些局限:(1) 秩约束的骨干网络微调,注入低秩投影残差以促使学习解耦的内容子空间与风格子空间;(2) 一种提示引导方法,采用带有专门分支的专家编码器,通过选择性适配器聚合实现语义扩展与精确控制;(3) 一种无训练的、依赖时间步的无分类器引导方案,通过在扩散步骤间策略性地调整噪声预测来增强生成稳定性。我们的方法显著提升了内容-风格解耦程度,实现了对 LoRA 模块组合的灵活语义控制,并在无需额外重训练开销的情况下达到高保真生成。
英文原文摘要 1368 字符
Personalized image generation requires effectively balancing content fidelity with stylistic consistency when synthesizing images based on text and reference examples. Low-Rank Adaptation (LoRA) offers an efficient personalization approach, with potential for precise control through combining LoRA weights on different concepts. However, existing combination techniques face persistent challenges: entanglement between content and style representations, insufficient guidance for controlling elements' influence, and unstable weight fusion that often require additional training. We address these limitations through CRAFT-LoRA, with complementary components: (1) rank-constrained backbone fine-tuning that injects low-rank projection residuals to encourage learning decoupled content and style subspaces; (2) a prompt-guided approach featuring an expert encoder with specialized branches that enables semantic extension and precise control through selective adapter aggregation; and (3) a training-free, timestep-dependent classifier-free guidance scheme that enhances generation stability by strategically adjusting noise predictions across diffusion steps. Our method significantly improves content-style disentanglement, enables flexible semantic control over LoRA module combinations, and achieves high-fidelity generation without additional retraining overhead.