面向大型视觉语言模型的注意力感知推理优化与显存高效解码
Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding
针对多图/视频长上下文 VLM 解码时 KV 缓存显存爆炸的问题，用隐式低秩结构压缩多头注意力的 K/V 矩阵，并配合 token 级注意力感知解压降低延迟；做视觉长上下文推理加速的工作可直接复用该 KV 压缩与解压调度思路，且能与 eviction、量化、kernel fusion 叠加。
原文依据 We introduce a multi-head attention compaction method for economically storing key and value matrices by exploiting the implicit low-rank structure
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（VLM）在多模态推理上取得了显著成功，但由于解码阶段的显存开销，其推理时间效率仍是一项重大挑战，尤其当 VLM 的查询与回答由长序列的视觉与文本 token 构成时。本文提出 AttentionPack，一个面向大型视觉语言模型的自适应、注意力感知优化框架，旨在提升解码阶段的显存效率，重点应对视觉输入与交互数量激增所带来的挑战，特别是涉及多张高分辨率图像或视频的长上下文任务。AttentionPack 的新颖之处有两点：(i) 提出多头注意力压缩方法，利用隐式低秩结构以更经济的方式存储键和值矩阵；(ii) 设计 token 特定的注意力感知解压机制以降低延迟开销。在多个基准上的实验结果表明，AttentionPack 将显存效率最高提升 8 倍，在保持模型输出质量的同时支持更大的批大小与更快的批量推理，或支持更长的上下文长度以获得更优的检索性能。我们还报告了 AttentionPack 与淘汰、量化及算子融合结合后的有效性，表明其在资源受限环境下可带来进一步的效率收益。
英文原文摘要 1420 字符
Large Vision-Language Models (VLMs) have achieved remarkable success in multi-modal reasoning, but their inference time efficiency remains a significant challenge due to the memory overhead during decoding, especially when the query and answer of VLMs consist of long sequences of visual and text tokens. This paper presents AttentionPack, an adaptive and attention-aware optimization framework tailored for large vision-language models with improving memory-efficiency during decoding, focusing on addressing the challenges due to the increased high number of visual inputs and interactions, particularly in long-context tasks with multiple high-resolution images or videos. AttentionPack is novel in two aspects: (i) We introduce a multi-head attention compaction method for economically storing key and value matrices by exploiting the implicit low-rank structure, and (ii) we develop a token-specific attention-aware decompression mechanism to reduce latency overhead. Experimental results on multiple benchmarks demonstrate that AttentionPack improves memory efficiency by up to 8x, enabling higher batch sizes and faster batch inference while preserving the model output quality or longer context lengths for superior retrieval performance. We also report the effectiveness of AttentionPack combined with eviction, quantization and kernel fusion, showing further efficiency gains for resource-limited environments.