HBridge：面向统一多模态理解与生成的异构专家 H 形桥接
HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation
HBridge 的非对称 H 形桥接结构只在中层共享注意力、浅层与深层解耦，为视觉研究者提供了一种可复用的异构专家融合范式：在保留 LLM 与扩散模型各自预训练先验的同时降低跨模态注意力的共享开销，可直接迁移到需要同时做视觉理解与图像生成的统一模型中。
原文依据 HBridge selectively bridges intermediate layers, reducing over 40% attention sharing
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期的统一模型将理解专家（如 LLM）与生成专家（如扩散模型）集成在一起，取得了很强的多模态性能。然而，BAGEL 和 LMFusion 等近期先进方法遵循混合 Transformer（Mixture-of-Transformers，MoT）范式，采用对称设计，将一个专家镜像到另一个专家，以便于初始化和融合；由于固有的模态差异，这种设计仍非最优。在本工作中，我们提出 HBridge，一种非对称的 H 形架构，使异构专家能够最优地利用各自模态领域的预训练先验。与以往通过共享注意力直接连接专家之间所有层的稠密融合策略不同，HBridge 选择性地桥接中间层，减少了超过 40% 的注意力共享，从而提升效率并改善生成质量。捕获模态特定表示的浅层与深层被解耦，而中层桥接则促进语义对齐。为进一步增强跨模态一致性，我们引入语义重建 token，显式引导生成专家重建目标图像的视觉语义 token。在多个基准上的大量实验证明了 HBridge 的有效性与优越性能，为统一多模态生成确立了新范式。
英文原文摘要 1838 字符
Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models), achieving strong multimodal performance. However, recent advanced methods such as BAGEL and LMFusion follow the Mixture-of-Transformers (MoT) paradigm, adopting a symmetric design that mirrors one expert to another for convenient initialization and fusion, which remains suboptimal due to inherent modality discrepancies. In this work, we propose HBridge, an asymmetric H-shaped architecture that enables heterogeneous experts to optimally leverage pretrained priors from their respective Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models), achieving strong multimodal performance. However, recent advanced methods such as BAGEL and LMFusion follow the Mixture-of-Transformers (MoT) paradigm, adopting a symmetric design that mirrors one expert to another for convenient initialization and fusion, which remains suboptimal due to inherent modality discrepancies. modality domains. Unlike prior dense fusion strategies that straightforwardly connect all layers between experts via shared attention, HBridge selectively bridges intermediate layers, reducing over 40% attention sharing, which improves efficiency and enhances generation quality. Shallow and deep layers, which capture modality-specific representations, are decoupled, while mid-layer bridging promotes semantic alignment. To further strengthen cross-modal coherence, we introduce semantic reconstruction tokens that explicitly guide the generative expert to reconstruct visual semantic tokens of the target image. Extensive experiments across multiple benchmarks demonstrate the effectiveness and superior performance of HBridge, establishing a new paradigm for unified multimodal generation.