基于局部与全局上下文优化的词元缩减：面向高效视频大语言模型
Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models
面向视频大语言模型的免训练词元压缩：用注意力引导的帧内/帧间锚点加最优传输，把被剪词元的信息聚合进锚点，可直接套用到任何视频 LLM 的视觉编码器与 LLM 之间，降低视觉词元数而不丢时序细节。
原文依据 we propose a new perspective that elaborates token Anchors within intra-frame and inter-frame to comprehensively aggregate the informative contexts via local-global Optimal Transport (AOT)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频大语言模型（VLLM）展现出强大的视频理解能力，但由于视觉词元冗余而存在效率低下的问题。现有剪枝方法主要针对帧内空间冗余，或在 LLM 内部进行剪枝并带来浅层开销，导致时空缩减效果欠佳，且未能充分利用长上下文的可压缩性。这些方法往往还会丢弃被合并或剪枝词元中细微但有用的上下文信息。本文提出一种新视角：在帧内与帧间构建词元锚点，通过局部-全局最优传输（AOT）全面聚合有用上下文。具体而言，我们首先在注意力引导下于每帧内建立局部与全局感知的词元锚点，随后由最优传输从被剪枝词元中聚合有用上下文，构建帧内词元锚点。接着，基于时序帧片段，将每个片段内的首帧作为关键帧锚点，通过最优传输汇聚连续帧中的相似信息，同时保留具有区分性的词元以表征时序动态，从而以免训练的方式实现高效的词元缩减。大量评估表明，所提出的 AOT 在领先的视频 LLM 上于多种短、长视频基准中取得有竞争力的性能，在保持时序与视觉保真度的同时获得显著的计算效率。
英文原文摘要 1459 字符
Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens. Existing pruning primary targets intra-frame spatial redundancy or prunes inside the LLM with shallow-layer overhead, yielding suboptimal spatiotemporal reduction and underutilizing long-context compressibility. All of them often discard subtle yet informative context from merged or pruned tokens. In this paper, we propose a new perspective that elaborates token Anchors within intra-frame and inter-frame to comprehensively aggregate the informative contexts via local-global Optimal Transport (AOT). Specifically, we first establish local- and global-aware token anchors within each frame under the attention guidance, which then optimal transport aggregates the informative contexts from pruned tokens, constructing intra-frame token anchors. Then, building on the temporal frame clips, the first frame within each clip will be considered as the keyframe anchors to ensemble similar information from consecutive frames through optimal transport, while keeping distinct tokens to represent temporal dynamics, leading to efficient token reduction in a training-free manner. Extensive evaluations show that our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs, obtaining substantial computational efficiency while preserving temporal and visual fidelity.