ApET：面向高效视觉语言模型的近似误差引导词元压缩
ApET: Approximation-Error Guided Token Compression for Efficient VLMs
用线性近似重构误差衡量视觉词元信息量并据此剪枝，摆脱对 [CLS] 注意力或文本-视觉交叉注意力的依赖，从而兼容 FlashAttention，可直接用于任意 VLM 的视觉编码器输出端做免训练或轻量加速。
原文依据 ApET first reconstructs the original visual tokens with a small set of basis tokens via linear approximation, then leverages the approximation error to identify and drop the least informative tokens.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期的视觉语言模型（VLM）已展现出卓越的多模态理解能力，但冗余的视觉词元带来了高昂的计算开销并降低了推理效率。已有研究通常依赖 [CLS] 注意力或文本-视觉交叉注意力来识别并丢弃冗余视觉词元。尽管结果可观，这类方案容易引入位置偏置，更关键的是与 FlashAttention 等高效注意力内核不兼容，限制了其在 VLM 加速中的实际部署。本文摆脱对注意力的依赖，从信息论视角重新审视视觉词元压缩，目标是在完全不涉及注意力的情况下最大程度保留视觉信息。我们提出 ApET，一种近似误差引导的词元压缩框架。ApET 首先通过线性近似用一小组基词元重构原始视觉词元，然后利用近似误差识别并丢弃信息量最少的词元。在多个 VLM 与基准上的大量实验表明，ApET 在图像理解任务上保留了原始性能的 95.2%，在视频理解任务上甚至达到 100.4%，同时分别将词元预算压缩了 88.9% 和 87.5%。得益于其无注意力设计，ApET 可无缝集成 FlashAttention，从而实现进一步的推理加速，使 VLM 部署更加实用。
英文原文摘要 1506 字符
Recent Vision-Language Models (VLMs) have demonstrated remarkable multimodal understanding capabilities, yet the redundant visual tokens incur prohibitive computational overhead and degrade inference efficiency. Prior studies typically relies on [CLS] attention or text-vision cross-attention to identify and discard redundant visual tokens. Despite promising results, such solutions are prone to introduce positional bias and, more critically, are incompatible with efficient attention kernels such as FlashAttention, limiting their practical deployment for VLM acceleration. In this paper, we step away from attention dependencies and revisit visual token compression from an information-theoretic perspective, aiming to maximally preserve visual information without any attention involvement. We present ApET, an Approximation-Error Token compression framework. ApET first reconstructs the original visual tokens with a small set of basis tokens via linear approximation, then leverages the approximation error to identify and drop the least informative tokens. Extensive experiments across multiple VLMs and benchmarks demonstrate that ApET retains 95.2% of the original performance on image-understanding tasks and even attains 100.4% on video-understanding tasks, while compressing the token budgets by 88.9% and 87.5%, respectively. Thanks to its attention-free design, ApET seamlessly integrates with FlashAttention, enabling further inference acceleration and making VLM deployment more practical.