CoIn：面向高效大型多模态模型的覆盖度与信息量引导词元削减
CoIn: Coverage and Informativeness-Guided Token Reduction for Efficient Large Multimodal Models
把视觉词元裁剪建模为「信息量 + 覆盖度」的联合最优子集选择，用基于体积的覆盖度准则保证特征空间全局代表性，且与 kernel 级推理加速器兼容——做 LMM 推理加速或多模态 token 压缩时可直接替换 importance-only 或 pairwise 去冗余的选 token 策略。
原文依据 we reformulate visual token reduction as an optimal subset selection problem jointly guided by two complementary objectives: informativeness and coverage
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型多模态模型（LMM）在视觉理解任务上取得了显著成功。LMM 将视觉与文本输入编码为词元，随后交由大型语言模型（LLM）处理。然而，数量庞大的视觉词元成为推理效率与内存占用的主要瓶颈。削减视觉词元是一种颇具前景的免训练方案，但现有方法仍存在局限。基于重要性的方法泛化性差、与内核级推理优化不兼容，且只考虑单一模态的信息。基于多样性的策略通常只关注词元两两之间的冗余，并把所有词元视为同等重要。近期尝试将重要性与多样性准则顺序结合的方法，仍无法解决其底层度量指标的内在缺陷。为克服这些局限，我们将视觉词元削减重新表述为一个由两个互补目标——信息量与覆盖度——联合引导的最优子集选择问题。信息量通过每个词元的内在显著性与视觉-文本对齐程度来量化，覆盖度则通过基于体积的子集选择准则来施加，以确保在视觉特征空间中的全局代表性。这一联合表述在端到端的词元选择过程中有效整合了视觉显著性、跨模态对齐与全局覆盖度，形成一个计算高效、模型无关且与现代推理加速器兼容的框架。大量实验表明，CoIn 在保持强劲任务性能的同时大幅降低了计算与内存开销。
英文原文摘要 1667 字符
Large Multimodal Models (LMMs) have shown remarkable success in visual understanding tasks. LMMs encode visual and textual inputs into tokens, which are then processed by Large Language Models (LLMs). However, the large number of visual tokens poses a major bottleneck for inference efficiency and memory usage. Reducing visual tokens is a promising training-free solution, but existing methods remain limited. Importance-based approaches suffer from poor generalization, are incompatible with kernel-level inference optimizations, and only consider information from a single modality. Diversity-based strategies typically focus on pairwise token redundancy and treat all tokens as equally important. Recent attempts to sequentially combine importance and diversity criteria still fail to address the intrinsic drawbacks of their underlying metrics. To address these limitations, we reformulate visual token reduction as an optimal subset selection problem jointly guided by two complementary objectives: informativeness and coverage. Informativeness is quantified through per-token intrinsic saliency and visual-textual alignment, while coverage is enforced via a volume-based subset selection criterion that ensures global representativeness in the visual feature space. This joint formulation effectively integrates visual saliency, cross-modal alignment, and global coverage in an end-to-end token selection process, yielding a computationally efficient, model-agnostic framework compatible with modern inference accelerators. Extensive experiments demonstrate that CoIn substantially reduces computation and memory cost while maintaining strong task performance.