面向超低保留率视频大语言模型的统一时空词元压缩
Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention
提出一种无需重训练、即插即用的视频大语言模型视觉词元压缩方法，通过全局时空分配和文本感知合并，在极低保留率下保持性能，可直接用于降低视频理解任务的推理成本。
原文依据 Without requiring retraining, our method serves as a plug-and-play module compatible with existing Video-LLMs.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频大语言模型（Video-LLMs）由于大量视觉词元而面临高昂的计算成本。现有的词元压缩方法通常采用两阶段时空压缩策略，依赖阶段特定的度量指标，并隐含时空可分离的假设。然而，在极低的保留率下，这类方法往往导致分配不均衡，并丢失对问答至关重要的视觉证据。我们将词元压缩重新表述为全局词元保留池内的时空分配任务。我们提出了一种统一的选择机制，融合注意力权重与语义相似度，全局选择高贡献且低冗余的词元。未被选中的词元通过聚类进行合并并回填，以保持信息完整性。在LLM内部，我们进一步引入文本感知合并，基于查询相关性进行二次压缩。无需重新训练，我们的方法可作为即插即用模块与现有Video-LLMs兼容。实验表明，仅保留约2%的视觉词元，即可在多个基准上保持90.1%的基线性能，同时将浮点运算量降至约2.6%。这些优势可泛化到多种骨干网络，降低端到端推理延迟和内存消耗。我们的统一时空词元压缩策略在超低词元保留率下的视频理解中达到了最优水平。
英文原文摘要 1486 字符
Video large language models (Video-LLMs) face high computational costs due to large volumes of visual tokens. Existing token compression methods typically adopt a two-stage spatiotemporal compression strategy, relying on stage-specific metrics and an implicit assumption of spatiotemporal separability. Under extremely low retention ratios, however, such approaches often result in unbalanced allocation and loss of visual evidence essential for question answering. We reformulate token compression as a spatiotemporal allocation task within a global token retention pool. We propose a unified selection mechanism that integrates attention weights and semantic similarity to globally select tokens with high contribution and low redundancy. Unselected tokens are merged via clustering and refilled, preserving information integrity. Inside the LLM, we further introduce text-aware merging to perform secondary compression based on query relevance. Without requiring retraining, our method serves as a plug-and-play module compatible with existing Video-LLMs. Experiments show that retaining only about 2% of visual tokens preserves 90.1% of baseline performance across multiple benchmarks, while reducing FLOPs to roughly 2.6%. These benefits generalize across diverse backbones, decreasing end-to-end inference latency and memory consumption. Our unified spatiotemporal token compression strategy establishes the state-of-the-art in video understanding under ultra-low token retention.