释放多模态大语言模型内在的视觉表征能力
Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
把掩码图像建模搬到 MLLM 的联合潜语义空间里做视觉重建，为视觉表征提供直接监督信号，可复用于任何需要增强 VLM 密集视觉能力的训练流程。
原文依据 learning more discriminative visual representations via masked image modeling in the joint latent semantic space of LLM
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLM）在多模态任务中已展现出卓越的能力。尽管性能令人印象深刻，MLLM 仍受困于模态失衡问题：在较深层中，视觉信息相较文本表征往往未被充分利用，导致视觉性能下降或产生幻觉。该问题源于训练过程中主要依赖下一文本词元预测，这种方式无法提供直接的视觉监督信号，使得视觉表征在各层中逐渐同质化。为此，我们提出潜空间视觉重建（LaVer），一种新的训练框架，通过在 LLM 联合潜语义空间中进行掩码图像建模，促使 MLLM 学习更具判别性的视觉表征。我们的方法为 MLLM 提供直接的视觉激活，使其表现出更高的视觉注意力分配，表明视觉信息的利用得到增强。在多种基准上的大量实验证明了该方法在各类场景下的优越性，尤其是在需要密集视觉能力的场景中。代码将在论文发表后公开。
英文原文摘要 1192 字符
Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in multimodal tasks.Despite their impressive performance, MLLMs suffer from the modality imbalance issue, where visual information is often underutilized compared to textual representations in deeper layers, leading to degraded visual performance or hallucinations.This issue stems from the predominant reliance on next-text-token-prediction during training, which fails to provide direct visual supervisory signals, resulting in progressive homogenization of visual representations throughout the layers.To this end, we propose Latent Visual Reconstruction (LaVer), a novel training framework that facilitates MLLMs in learning more discriminative visual representations via masked image modeling in the joint latent semantic space of LLM.Our method offers direct visual activation to MLLMs, which exhibit increased visual attention allocation, indicating enhanced utilization of visual information.Extensive experiments across diverse benchmarks prove the superiority of our approach in various scenarios, especially those requiring dense visual capabilities.Code will be publicly available upon publication.