OmniZip：面向快速全模态大模型的语言模型音频引导动态词元压缩
OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models
对做视频多模态理解的人，这套「用音频显著性动态决定视频词元保留率」的免训练压缩流程可以直接插到任意 OmniLLM 的视觉编码器之后，作为长视频词元预算分配的可复用策略；其按时间组计算信息密度再交错时空剪枝的做法，也可迁移到纯视觉视频 LLM 的帧级词元裁剪。
原文依据 OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information density
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论全模态大语言模型（OmniLLM）近来在统一音视频理解方向受到越来越多的研究关注。然而，处理较长的联合音视频词元序列所带来的高计算开销已成为关键瓶颈。现有的词元压缩方法尚未回应联合压缩多模态词元这一新出现的需求。为填补这一空白，本文提出 OmniZip，一种免训练、由音频引导的音视频词元压缩框架，用于优化多模态词元表示并加速模型推理。具体而言，OmniZip 首先识别显著的音频词元，随后为每个时间组计算音频保留分数以刻画信息密度，从而动态引导视频词元剪枝，并借助跨模态相似度增强来保留来自音频锚点的线索。对于每个时间窗口，OmniZip 采用交错的时空方案压缩视频词元。大量实验结果表明了 OmniZip 的优势：与其他表现最佳的同类方法相比，它在无需训练的情况下实现了 3.42 倍的推理加速和 1.4 倍的内存降低，同时保持了 OmniLLM 的性能。
英文原文摘要 1162 字符
Omnimodal large language models (OmniLLMs) have attracted increasing research attention of late towards unified audio-video understanding. However, the high computational cost of processing longer joint audio-video token sequences has become a key bottleneck. Existing token compression methods have not addressed the emerging need to jointly compress multimodal tokens. To bridge this gap, we present OmniZip, a training-free, audio-guided audio-visual token-compression framework that optimizes multimodal token representation and accelerates model inference. Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information density, thereby dynamically guiding video token pruning and preserving cues from audio anchors enhanced by cross-modal similarity. For each time window, OmniZip compresses the video tokens using an interleaved spatio-temporal scheme. Extensive results demonstrate the merits of OmniZip: it achieves a 3.42x inference speedup and a 1.4x memory reduction over other top-performing counterparts, while maintaining the performance of OmniLLMs without training.