无需多模态注意力的门控条件注入:面向可控线性注意力 Transformer
Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformers
用统一门控条件模块替代多模态注意力,在 SANA 等线性注意力骨干上注入空间对齐与非对齐条件,为端侧可控扩散生成提供可复用的条件注入范式。
原文依据 a unified gated conditioning module working in a dual-path pipeline, which effectively integrates multi-type conditional inputs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期基于扩散的可控视觉生成进展使图像质量获得显著提升。然而,这些强大的模型由于计算需求庞大,通常部署在云端服务器上,从而引发了对用户数据隐私的严重担忧。为实现安全高效的端侧生成,本文探索基于线性注意力架构构建的可控扩散模型,这类架构即使在边缘设备上也具备优越的可扩展性与效率。然而,我们的实验表明,现有的可控生成框架(如 ControlNet 与 OminiControl)要么缺乏支持多种异构条件类型的灵活性,要么在此类线性注意力模型上收敛缓慢。为解决这些局限,我们提出一种面向 SANA 等线性注意力骨干网络的新型可控扩散框架。方法的核心在于一个以双路径流水线工作的统一门控条件模块,它能够有效整合多类型条件输入,例如空间对齐与非对齐的线索。在多个任务与基准上的大量实验表明,我们的方法在线性注意力模型上实现了最优的可控生成性能,在保真度与可控性方面均超越现有方法。
英文原文摘要 1354 字符
Recent advances in diffusion-based controllable visual generation have led to remarkable improvements in image quality. However, these powerful models are typically deployed on cloud servers due to their large computational demands, raising serious concerns about user data privacy. To enable secure and efficient on-device generation, we explore in this paper controllable diffusion models built upon linear attention architectures, which offer superior scalability and efficiency, even on edge devices. Yet, our experiments reveal that existing controllable generation frameworks, such as ControlNet and OminiControl, either lack the flexibility to support multiple heterogeneous condition types or suffer from slow convergence on such linear-attention models. To address these limitations, we propose a novel controllable diffusion framework tailored for linear attention backbones like SANA. The core of our method lies in a unified gated conditioning module working in a dual-path pipeline, which effectively integrates multi-type conditional inputs, such as spatially aligned and non-aligned cues. Extensive experiments on multiple tasks and benchmarks demonstrate that our approach achieves state-of-the-art controllable generation performance based on linear-attention models, surpassing existing methods in terms of fidelity and controllability.