MMFace-DiT：用于高保真多模态人脸生成的双流扩散 Transformer
MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation
双流 Transformer 块用共享 RoPE 注意力并行处理空间 token（掩码/草图）与语义 token（文本），避免模态主导，可直接迁移到任何需要同时满足文本语义与结构先验的可控生成任务；Modality Embedder 让单一模型免重训适配不同空间条件，是统一多条件控制的可复用思路。
原文依据 a dual-stream transformer block that processes spatial (mask/sketch) and semantic (text) tokens in parallel, deeply fusing them through a shared Rotary Position-Embedded (RoPE) Attention mechanism
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期的多模态人脸生成模型通过为基于文本的条件补充分割掩码、草图或边缘图等空间先验，缓解了文本到图像扩散模型在空间控制上的局限。这种多模态融合使得合成结果既能对齐高层语义意图，又能对齐低层结构布局。然而，大多数现有方法通常通过附加辅助控制模块或拼接彼此独立的单模态网络来扩展预训练的文本到图像流水线。这类临时性设计继承了架构约束、造成参数冗余，并且在模态冲突或潜变量空间不匹配时往往失效，限制了其在语义域与空间域之间进行协同融合的能力。我们提出 MMFace-DiT，一个为协同多模态人脸合成而设计的统一双流扩散 Transformer。其核心创新在于一个双流 Transformer 块，它并行处理空间 token（掩码/草图）与语义 token（文本），并通过共享的旋转位置编码（RoPE）注意力机制将二者深度融合。该设计可防止模态主导，确保对文本与结构先验的强遵循，从而在可控人脸生成中实现前所未有的空间—语义一致性。此外，一种新颖的模态嵌入器使单一内聚模型无需重新训练即可动态适配不同的空间条件。相比六个最优的多模态人脸生成模型，MMFace-DiT 在视觉保真度与提示词对齐上取得 40% 的提升，为端到端可控生成建模建立了一种灵活的新范式。
英文原文摘要 1676 字符
Recent multimodal face generation models address the spatial control limitations of text-to-image diffusion models by augmenting text-based conditioning with spatial priors such as segmentation masks, sketches, or edge maps. This multimodal fusion enables controllable synthesis aligned with both high-level semantic intent and low-level structural layout. However, most existing approaches typically extend pre-trained text-to-image pipelines by appending auxiliary control modules or stitching together separate uni-modal networks. These ad hoc designs inherit architectural constraints, duplicate parameters, and often fail under conflicting modalities or mismatched latent spaces, limiting their ability to perform synergistic fusion across semantic and spatial domains. We introduce MMFace-DiT, a unified dual-stream diffusion transformer engineered for synergistic multimodal face synthesis. Its core novelty lies in a dual-stream transformer block that processes spatial (mask/sketch) and semantic (text) tokens in parallel, deeply fusing them through a shared Rotary Position-Embedded (RoPE) Attention mechanism. This design prevents modal dominance and ensures strong adherence to both text and structural priors to achieve unprecedented spatial-semantic consistency for controllable face generation. Furthermore, a novel Modality Embedder enables a single cohesive model to dynamically adapt to varying spatial conditions without retraining. MMFace-DiT achieves a 40% improvement in visual fidelity and prompt alignment over six state-of-the-art multimodal face generation models, establishing a flexible new paradigm for end-to-end controllable generative modeling.