自我提示:唤醒一维视觉分词器中的文本语义
Prompt Yourself: Awakening Textual Semantics in 1D Visual Tokenizers
提出一种自提示训练范式,在训练时同时生成视觉与文本词元并对齐到预训练语言模型,从而将语言语义注入1D视觉分词器,提升细粒度图像编码能力;该思路可复用于改进其他视觉分词器或生成模型的表征质量。
原文依据 During training, VLTok simultaneously generates 1D visual and textual tokens from images, aligning the textual tokens with embeddings from a pre-trained language model.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论一维(1D)视觉分词器通过丢弃局部空间先验,实现了显著的语义紧凑性,在图像重建与生成任务中日益流行。然而,这种全局且序列化的表示难以保留细粒度的视觉内容;单纯增加网络规模或词元数量只能带来表面上的缓解。为此,我们提出VLTok,一种新颖的一维混合分词器,通过自提示训练范式在共享词元空间中统一视觉与语言表示。训练时,VLTok同时从图像生成一维视觉词元和文本词元,并将文本词元与预训练语言模型的嵌入对齐。这种跨模态对齐将隐式语言线索注入分词器,增强细粒度图像编码能力。推理时,自提示范式无需外部文本,保持了仅图像框架的简洁性,同时受益于多模态引导。在ImageNet基准上的大量实验表明,VLTok在图像重建和图像生成方面均达到最优性能。例如,在相同模型参数预算下,相比GigaTok,我们的方法在rFID上相对降低11.1%,在gFID上相对降低18.7%。
英文原文摘要 1330 字符
One-dimensional (1D) visual tokenizers offer notable semantic compactness by discarding local spatial priors, and have become increasingly popular for image reconstruction and generation tasks. However, such global and sequential representations struggle to preserve fine-grained visual content; simply increasing network size or token count offers only superficial mitigation. To address this, we introduce VLTok, a novel 1D hybrid tokenizer that unifies visual and language representations in a shared token space through a self-prompted training paradigm. During training, VLTok simultaneously generates 1D visual and textual tokens from images, aligning the textual tokens with embeddings from a pre-trained language model. This cross-modal alignment infuses implicit linguistic cues into the tokenizer, enhancing fine-grained image encoding. At inference, the self-prompted paradigm eliminates the need for external text, maintaining the simplicity of the image-only framework while benefiting from multi-modal guidance. Extensive experiments on the ImageNet benchmark demonstrate that VLTok achieves state-of-the-art performance in both image reconstruction and image generation. For example, under the same model parameter budget, our method yields relative reduction of 11.1% in rFID and 18.7% in gFID compared to GigaTok.