探索面向语言智能学习的视觉预训练
Exploring Visual Pretraining for Learning Language Intelligence
把文档图像当作语言模型的预训练语料，用掩码自回归目标让 LLM 从可见区域推断被遮挡区域的潜变量假设，为视觉研究者提供了一个可复用的思路：用视觉输入替代或补充文本语料来提升语言推理能力，并可用潜空间对齐分析跨模态表征。
原文依据 leverages raw document images to improve language intelligence
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管最基础的预训练范式通常是在各自的数据集上训练模态专属模型，但「柏拉图式表征假说」认为，随着数据和模型规模的增长，不同模态的表征最终会趋于对齐；这一假说暗示了一种耐人寻味的可能性：大语言模型（LLM）可以在视觉语料上预训练，从而达到与文本预训练模型相当的水平，从而扩展数据来源以突破文本规模瓶颈，并利用更丰富的视觉线索实现更全面的语料理解。本文首次尝试验证这一推论是否可行，提出了面向语言智能学习的掩码自回归预训练（MAPLE），这是一种面向 LLM 的新型视觉预训练范式，利用原始文档图像来提升语言智能。MAPLE 具有通用性，可将掩码自回归模型与多种 LLM 主干网络集成，其中 LLM 被激励根据未掩码区域为被掩码区域生成潜变量假设。我们在数学推理领域用多种 LLM 主干网络验证了 MAPLE，结果表明，在四个数学推理基准上，MAPLE 的平均准确率相对纯文本预训练始终更优，最高提升 40.2%。进一步分析表明，经过视觉预训练的 LLM 学到了一个共享潜空间，将文档视觉信息与文本对齐，并利用了版面和结构线索，这支持了视觉预训练作为通往更强语言模型的一条可行且可扩展的路径。
英文原文摘要 1514 字符
While the most fundamental pretraining paradigm typically trains modality-specific models on their respective datasets, the Platonic Representation Hypothesis that representations eventually align across modalities as data and model scale suggests an intriguing possibility: large language models (LLMs) could be pretrained on visual corpora to reach parity with text-pretrained models, thereby expanding data sources to break the text-scaling bottlenecks, and leveraging richer visual cues for more comprehensive corpus understanding. This paper makes the first attempt to demonstrate the feasibility of this implication by introducing Masked Autoregressive Pretraining for Learning language intelligencE (MAPLE), a novel visual pretraining paradigm for LLMs that leverages raw document images to improve language intelligence. MAPLE is universal to integrate masked auto-regressive models with various LLM backbones, where the LLMs are incentivized to generate latent hypotheses for the masked regions based on the unmasked regions. We verify MAPLE in the domain of math reasoning with multiple LLM backbones and show that MAPLE consistently surpasses text-only pretraining relatively by at most 40.2% on average accuracy across four math reasoning benchmarks. Further analyses show that visually pretrained LLMs learn a shared latent space that aligns document visuals with text and exploits layout and structural cues, supporting visual pretraining as a feasible and scalable route to stronger language models.