按需视觉：以稀疏、动态选择的视觉-语言交互提升 VLLM 效率
VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions
对做多模态视觉研究的人，这篇论文给出了一条不丢弃视觉词元、而是稀疏化图文交互的加速路径：用少量交叉注意力层提供全局视觉上下文，再用动态选择的自注意力层按样本复杂度精炼高分辨率视觉表示，可直接迁移到任何 LVLM 的推理加速与高分辨率细粒度理解场景。
原文依据 VISOR improves efficiency by sparsifying the interaction between image and text tokens
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有提升大型视觉语言模型（LVLM）效率的方法大多基于视觉词元缩减这一思路。然而，这种做法会造成信息瓶颈，损害模型性能，在需要细粒度理解与推理的困难任务上尤为明显。在本工作中，我们挑战这一范式，提出按需视觉（VISion On Request，VISOR），一种在不丢弃视觉信息的前提下降低推理开销的方法。VISOR 不压缩图像，而是通过稀疏化图像词元与文本词元之间的交互来提升效率。具体而言，语言模型通过一小组经过策略性放置的注意力层来关注完整的高分辨率视觉词元集合：通用的视觉上下文由文本与图像之间高效的交叉注意力提供，而少数放置得当且被动态选择的自注意力层则对视觉表示本身进行精炼，从而在需要时实现复杂的高分辨率推理。基于这一原则，我们首先通过改变自注意力层的数量，在一系列计算预算上训练一个单一的通用网络，随后引入一种轻量的策略机制，根据每个样本的复杂度动态分配视觉计算量。大量实验表明，VISOR 在显著降低计算成本的同时，在多样化的基准测试套件上达到或超过最优结果，并在需要细致视觉理解的困难任务上表现突出。
英文原文摘要 1534 字符
Existing approaches for improving the efficiency of Large Vision-Language Models (LVLMs) are largely based on the concept of visual token reduction. This approach, however, creates an information bottleneck that impairs performance, especially on challenging tasks that require fine-grained understanding and reasoning. In this work, we challenge this paradigm by introducing VISion On Request (VISOR), a method that reduces inference cost without discarding visual information. Instead of compressing the image, VISOR improves efficiency by sparsifying the interaction between image and text tokens. Specifically, the language model attends to the full set of high-resolution visual tokens through a small, strategically placed set of attention layers: general visual context is provided by efficient cross-attention between text-image, while a few well-placed and dynamically selected self-attention layers refine the visual representations themselves, enabling complex, high-resolution reasoning when needed. Based on this principle, we first train a single universal network on a range of computational budgets by varying the number of self-attention layers, and then introduce a lightweight policy mechanism that dynamically allocates visual computation based on per-sample complexity. Extensive experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results across a diverse suite of benchmarks, and excels in challenging tasks that require detailed visual understanding.