Proxy-Tuning：为多模态自回归模型定制主体驱动图像生成能力
Proxy-Tuning: Tailoring Multimodal Autoregressive Models for Subject-Driven Image Generation
用扩散模型作为「代理监督」微调多模态自回归模型，使其获得主体驱动生成能力，并揭示弱到强泛化现象；对做个性化生成的人，这是一条绕开扩散架构、把 AR 模型拉进主体定制任务的可行路径，也提示可用强模型监督弱模型来补齐架构短板。
原文依据 we introduce Proxy-Tuning, leveraging diffusion models to enhance AR models' capabilities in subject-specific image generation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于下一词元预测与 Transformer 架构的多模态自回归（AR）模型，在包括文本到图像（T2I）生成在内的多种多模态任务中展现出卓越能力。尽管这类模型在通用 T2I 任务上表现强劲，我们的研究发现，与占据主导地位的扩散模型相比，它们在主体驱动图像生成上起初表现不佳。为弥补这一局限，我们提出 Proxy-Tuning，借助扩散模型来增强 AR 模型在特定主体图像生成方面的能力。我们的方法揭示了一个显著的弱到强现象：经过微调的 AR 模型在主体保真度与提示词遵循度两方面均持续优于其扩散模型监督者。我们分析了这一性能转变，并识别出 AR 模型表现更优的场景，尤其是在多主体组合与上下文理解方面。这项工作不仅在主体驱动的 AR 图像生成上取得了令人印象深刻的结果，还揭示了弱到强泛化在图像生成领域的潜力，有助于更深入地理解不同架构的优势与局限。
英文原文摘要 1172 字符
Multimodal autoregressive (AR) models, based on next-token prediction and transformer architecture, have demonstrated remarkable capabilities in various multimodal tasks including text-to-image (T2I) generation. Despite their strong performance in general T2I tasks, our research reveals that these models initially struggle with subject-driven image generation compared to dominant diffusion models. To address this limitation, we introduce Proxy-Tuning, leveraging diffusion models to enhance AR models' capabilities in subject-specific image generation. Our method reveals a striking weak-to-strong phenomenon: fine-tuned AR models consistently outperform their diffusion model supervisors in both subject fidelity and prompt adherence. We analyze this performance shift and identify scenarios where AR models excel, particularly in multi-subject compositions and contextual understanding. This work not only demonstrates impressive results in subject-driven AR image generation, but also unveils the potential of weak-to-strong generalization in the image generation domain, contributing to a deeper understanding of different architectures' strengths and limitations.