LLaDA-V：带视觉指令微调的大语言扩散模型
LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
把自回归解码换成掩码扩散式解码，并利用双向注意力建模视觉特征的空间关系，为视觉-语言模型提供了一条非自回归的架构路线，可直接复用其视觉编码器+MLP连接器+扩散语言模型的组合范式。
原文依据 leveraging diffusion language models' bidirectional attention to capture spatial relationships in visual data more effectively than causal, sequential processing
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文提出 LLaDA-V，一种纯基于扩散的多模态大语言模型（MLLM），它将视觉指令微调与掩码扩散模型相结合，偏离了当前多模态方法中占主导地位的自回归范式。LLaDA-V 构建在代表性大语言扩散模型 LLaDA 之上，引入视觉编码器与 MLP 连接器，将视觉特征投影到语言嵌入空间，并利用扩散语言模型的双向注意力，比因果式、序列式处理更有效地捕捉视觉数据中的空间关系。我们的实证研究揭示了若干有趣的结果：第一，尽管其语言模型在纯文本任务上弱于 LLaMA3-8B 和 Qwen2-7B 等对手，LLaDA-V 仍展现出有前景的多模态性能。在相同指令数据上训练时，LLaDA-V 在多模态任务上与 LLaMA3-V 高度相当，并在若干基准上表现出有前景的数据扩展行为。它还将与 Qwen2-VL 的性能差距缩小，表明其架构对多模态任务的有效性。第二，与现有纯基于扩散的 MLLM 相比，LLaDA-V 在多模态理解上取得了最优性能。我们的发现表明，大语言扩散模型在多模态语境中具有前景，值得进一步研究。为促进未来研究，我们在 https://github.com/ML-GSAI/LLaDA-V 开源了 LLaDA-V 及其训练与评测代码。
英文原文摘要 1558 字符
In this work, we introduce LLaDA-V, a purely diffusion-based Multimodal Large Language Model (MLLM) that integrates visual instruction tuning with masked diffusion models, representing a departure from the autoregressive paradigms dominant in current multimodal approaches. Built upon LLaDA, a representative large language diffusion model, LLaDA-V incorporates a vision encoder and MLP connector that projects visual features into the language embedding space, leveraging diffusion language models' bidirectional attention to capture spatial relationships in visual data more effectively than causal, sequential processing. Our empirical investigation reveals several intriguing results: First, LLaDA-V demonstrates promising multimodal performance despite its language model being weaker on purely textual tasks than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same instruction data, LLaDA-V is highly competitive with LLaMA3-V across multimodal tasks and shows promising data scaling behavior on several benchmarks. It also narrows the performance gap to Qwen2-VL, suggesting the effectiveness of its architecture for multimodal tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal understanding compared to existing purely diffusion-based MLLMs. Our findings suggest that large language diffusion models show promise in multimodal contexts and warrant further investigation. To facilitate future research, we open-source LLaDA-V together with its training and evaluation code at https://github.com/ML-GSAI/LLaDA-V.