UVU：通过视觉-语言统一自回归范式改进多模态理解
UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm
把像素级视觉重建作为预训练阶段的统一自回归监督信号，与文本 token 共享同一词表，为提升 MLLM 细粒度视觉感知提供了一条不依赖向量量化的连续视觉编码 + 码本构建思路，可直接迁移到其他多模态预训练配方中。
原文依据 we propose UVU, a novel vision-language unified autoregressive framework that eschews vector quantization
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管多模态大语言模型（MLLM）已取得显著进展，其细粒度视觉理解仍受限于对稀疏文本监督的主要依赖。现有引入视觉监督的工作通常在后训练阶段进行，此时视觉表征已基本固定，导致此类信号主要作为辅助约束，而非塑造感知特征的主要力量。本文旨在通过将视觉监督直接纳入预训练阶段，从根本上重塑模型的感知骨干网络。我们观察到，像素级图像块与文本 token 天然共存于一个共享的原始高维空间中，并具有内在的输入对称性。基于这一洞察，我们提出 UVU，一种新颖的视觉-语言统一自回归框架，它摒弃了向量量化。该框架独特地采用连续视觉编码以实现视觉输入的无损表示，并提出一种大规模迭代层次聚类算法来构建像素级视觉码本，从而扩展统一监督的词表，并支持像素级图像 token 与文本 token 的自回归生成。UVU 有效地将像素级视觉感知与语义级视觉理解协同起来，内化视觉重建能力，并在预训练阶段释放视觉监督对增强理解的促进作用。跨多个任务的大量实验表明，在 UVU 的监督学习范式下，MLLM 能够实现更优的多模态理解性能。
英文原文摘要 1707 字符
Despite remarkable advancements in multimodal large language models (MLLMs), their fine-grained visual understanding is constrained by a primary reliance on sparse textual supervision. Existing efforts to introduce visual supervision typically do so during post-training, when visual representations have already been largely fixed, causing such signals to act mainly as auxiliary constraints rather than as a primary force for shaping perceptual features. In this paper, we aim to fundamentally reshape the model's perceptual backbone by incorporating vision supervision directly into the pre-training stage. We observe that pixel-level image patches and textual tokens naturally coexist in a shared, raw high-dimensional space characterized by an inherent input symmetry. Leveraging this insight, we propose UVU, a novel vision-language unified autoregressive framework that eschews vector quantization. It uniquely employs continuous visual encoding for lossless representation of visual inputs and proposes a large-scale iterative hierarchical clustering algorithm to construct a pixel-level visual codebook, thereby extending the vocabulary for unified supervision and enabling autoregressive generation of pixel-level image tokens alongside textual tokens. UVU effectively synergizes pixel-level visual perception with semantic-level visual understanding, internalizing visual reconstruction capabilities and unlocking the facilitative role of visual supervision in enhancing understanding in the pre-training stage. Extensive experiments across multiple tasks demonstrate that MLLMs are capable of achieving superior multimodal understanding performance under the supervised learning paradigm of UVU.