重新思考大视觉语言模型的词元缩减
Rethinking Token Reduction for Large Vision-Language Models
将词元缩减从启发式剪枝/合并转为可学习的压缩映射，并针对多轮VQA中后续问题未知的设定设计提示词无关的压缩器，为LVLM推理加速提供可复用训练范式。
原文依据 we propose a learning-based prompt-agnostic method, termed MetaCompress, overcoming the limitations of heuristic designs.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大视觉语言模型（LVLM）在视觉理解与推理方面表现优异，但过多的视觉词元导致高昂的推理成本。尽管近期的词元缩减方法缓解了这一问题，但它们主要面向单轮视觉问答（VQA），对更实用的多轮VQA（MT-VQA）场景则基本未作探索。MT-VQA带来了额外挑战，因为后续问题事先未知且可能指向任意图像区域，使得现有缩减策略失效。具体而言，当前方法分为两类：依赖提示词的方法偏向初始文本提示词，丢弃了对后续轮次有用的信息；与提示词无关的方法虽然在技术上适用于多轮设定，但依赖注意力分数等启发式缩减指标，导致性能次优。本文提出一种基于学习的提示词无关方法MetaCompress，克服了启发式设计的局限。我们首先将词元缩减形式化为可学习的压缩映射，将剪枝和合并等现有形式统一为单一学习目标。在此形式化基础上，我们引入一种数据高效的训练范式，能够以有限的计算成本学习最优压缩映射。在MT-VQA基准和多种LVLM架构上的大量实验表明，MetaCompress在保持跨对话轮次强泛化的同时，实现了更优的效率-精度权衡。代码已开源：https://github.com/MArSha1147/MetaCompress。
英文原文摘要 1636 字符
Large Vision-Language Models (LVLMs) excel in visual understanding and reasoning, but the excessive visual tokens lead to high inference costs. Although recent token reduction methods mitigate this issue, they mainly target single-turn Visual Question Answering (VQA), leaving the more practical multi-turn VQA (MT-VQA) scenario largely unexplored. MT-VQA introduces additional challenges, as subsequent questions are unknown beforehand and may refer to arbitrary image regions, making existing reduction strategies ineffective. Specifically, current approaches fall into two categories: prompt-dependent methods, which bias toward the initial text prompt and discard information useful for subsequent turns; prompt-agnostic ones, which, though technically applicable to multi-turn settings, rely on heuristic reduction metrics such as attention scores, leading to suboptimal performance. In this paper, we propose a learning-based prompt-agnostic method, termed MetaCompress, overcoming the limitations of heuristic designs. We begin by formulating token reduction as a learnable compression mapping, unifying existing formats such as pruning and merging into a single learning objective. Upon this formulation, we introduce a data-efficient training paradigm capable of learning optimal compression mappings with limited computational costs. Extensive experiments on MT-VQA benchmarks and across multiple LVLM architectures demonstrate that MetaCompress achieves superior efficiency-accuracy trade-offs while maintaining strong generalization across dialogue turns. Our code is available at https://github.com/MArSha1147/MetaCompress.