视觉词元究竟编码了什么？揭示多模态大语言模型中的稀疏性与冗余性
What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
该论文揭示了多模态大语言模型中视觉词元的语义稀疏性（仅约60%的alive词元携带图像信息），并证明中层注入比初始嵌入更有效，为视觉编码器与LLM的高效对接提供了可复用的分析工具（EmbedLens）和设计原则（选择性词元剪枝、最小化视觉计算）。
原文依据 only the alive tokens, comprising ~60% of the total input, carry image-specific meaning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）将视觉词元投影到语言模型的嵌入空间，然而视觉语义的内部结构与处理机制仍鲜为人知。本文引入一个双重分析框架，其核心是一个新颖的探针工具EmbedLens，以进行细粒度分析。我们揭示了输入层面显著的语义稀疏性：视觉词元一致地划分为汇点、死区和活跃三类。值得注意的是，仅活跃词元（约占总输入的60%）携带图像特有的含义。此外，通过一个针对性的图像块压缩基准，我们证明这些活跃词元在进入LLM之前已编码了丰富的细粒度线索（如物体、颜色和OCR）。对于大多数标准任务，内部视觉计算（如视觉注意力和前馈网络）是冗余的。对于确实受益于内部处理的少数高度以视觉为中心的任务，我们发现活跃词元自然对齐于LLM的中间层而非初始嵌入空间，这表明浅层处理是不必要的，而直接中层注入既充分又有效。最终，我们的发现提供了视觉词元处理的统一机制视图，为通过选择性词元剪枝、最小化视觉计算和中层注入实现更高效、更可解释的MLLM架构铺平了道路。
英文原文摘要 1465 字符
Multimodal large language models (MLLMs) project visual tokens into the embedding space of language models, yet the internal structuring and processing of visual semantics remain poorly understood. In this work, we introduce a two-fold analytical framework featuring a novel probing tool, EmbedLens, to conduct a fine-grained analysis. We uncover a pronounced semantic sparsity at the input level: visual tokens consistently partition into sink, dead, and alive categories. Remarkably, only the alive tokens, comprising ~60% of the total input, carry image-specific meaning. Furthermore, using a targeted patch-compression benchmark, we demonstrate that these alive tokens already encode rich, fine-grained cues (e.g., objects, colors, and OCR) prior to entering the LLM. Internal visual computations (such as visual attention and feed-forward networks) are redundant for most standard tasks. For the small subset of highly vision-centric tasks that actually benefit from internal processing, we reveal that alive tokens naturally align with intermediate LLM layers rather than the initial embedding space, indicating that shallow-layer processing is unnecessary and that direct mid-layer injection is both sufficient. Ultimately, our findings provide a unified mechanistic view of visual token processing, paving the way for more efficient and interpretable MLLM architectures through selective token pruning, minimized visual computation, and mid-layer injection.