面向多模态大语言模型的更词元化的图像词元化方法
A More Word-like Image Tokenization for MLLMs
DiVT 将图像块嵌入聚类成语义单元，使视觉词元更接近离散词元，并动态调整词元预算，为视觉-语言模型提供了一种不修改编码器与语言模型即可降低视觉词元数量与延迟的即插即用方案。
原文依据 We propose a novel Disentangled Visual Tokenization (DiVT) that clusters patch embeddings into coherent semantic units
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现代多模态大语言模型通常保持语言模型固定，并训练一个视觉投影器，将像素映射为其嵌入空间中的词元序列，从而使图像能够以与文本基本相同的形态呈现。然而，语言模型是针对离散、语义有意义的词元进行优化的，而主流的视觉投影器将图像转换为长序列的连续且高度相关的嵌入。这导致视觉词元的行为与 LLM 最初训练时所理解的词元化单元不同。我们提出了一种新颖的解缠视觉词元化方法（DiVT），它将图像块嵌入聚类为连贯的语义单元，使每个词元对应一个独特的视觉概念，而非一个固定的网格单元。DiVT 进一步根据图像复杂度调整其词元预算，在不修改视觉编码器或语言模型的情况下，提供了明确的精度-计算权衡。在多样化的多模态基准上，DiVT 以显著更少的视觉词元达到或超越基线，在有限的词元预算下展现出鲁棒性，显著降低了内存成本和延迟，同时使视觉输入与 LLM 更加兼容。我们的代码可在 https://github.com/snuviplab/DiVT 获取。
英文原文摘要 1308 字符
Modern multimodal large language models (MLLMs) typically keep the language model fixed and train a visual projector that maps the pixels into a sequence of tokens in its embedding space, so that images can be presented in essentially the same form as text. However, the language model has been optimized to operate on discrete, semantically meaningful tokens, while prevailing visual projectors transform an image into a long stream of continuous and highly correlated embeddings. This causes the visual tokens to behave differently from the word-like units that LLMs are originally trained to understand. We propose a novel Disentangled Visual Tokenization (DiVT) that clusters patch embeddings into coherent semantic units, so each token corresponds to a distinct visual concept instead of a rigid grid cell. DiVT further adapts its token budget to image complexity, providing an explicit accuracy compute trade-off modifying neither the vision encoder nor the language model. Across diverse multimodal benchmarks, DiVT matches or surpasses baselines with significantly fewer visual tokens, demonstrating robustness under limited token budgets, significantly reducing memory cost and latency while making visual inputs more compatible with LLMs. Our code is available at https://github.com/snuviplab/DiVT.