DUET-VLM：面向视觉语言模型训练与推理的双阶段统一高效词元缩减
DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference
提供一种即插即用的双阶段视觉词元压缩方案：先在视觉编码器输出端做冗余感知合并，再在语言骨干网络内按层用文本显著性引导丢弃视觉词元，可直接嵌入 LLaVA/Video-LLaVA 类 VLM 的端到端训练与推理以大幅降低视觉词元数而几乎不掉点。
原文依据 a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）已具备出色的多模态理解与推理能力，但由于密集的视觉词元化，其计算开销依然高昂。现有高效方法要么合并冗余视觉词元，要么在语言骨干网络中逐步丢弃视觉词元，往往以精度换取速度。本文提出 DUET-VLM，一个通用的即插即用双阶段压缩框架，包含：(a) 仅基于视觉的冗余感知压缩，将视觉编码器的输出压缩为保留信息的词元；(b) 在语言骨干网络内按层进行由显著文本引导的视觉词元丢弃，以逐步剪除信息量较少的词元。这种协同的词元管理使得在保留关键语义的同时实现激进压缩。在 LLaVA-1.5-7B 上，本方法在词元数减少 67% 时仍保持基线精度 99% 以上，即使在减少 89% 时仍保持 97% 以上。借助训练中的双阶段压缩，它在 67% 和 89% 的压缩率下分别达到 99.7% 和 97.6% 的精度，在多个基准上超越此前最先进的视觉词元缩减方法。将其集成到 Video-LLaVA-7B 时，甚至超越基线——在词元大幅减少 53.1% 的情况下取得超过 100% 的精度，并在极端 93.4% 的设定下仍保持 97.6% 的精度。这些结果凸显了 DUET-VLM 端到端训练的价值，使其能够稳健地适应缩减后的视觉（图像/视频）输入而不牺牲精度，并在相同计算预算内产生紧凑而语义丰富的表征。代码已开源：https://github.com/AMD-AGI/DUET-VLM。
英文原文摘要 1622 字符
Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline--achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.