AdaptVision：通过自适应视觉获取实现高效视觉语言模型
AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition
把「视觉词元数量」当作可由模型自主决策的动作，用强化学习（DTPO）在粗到细的流程中决定何时调用裁剪工具获取高分辨率区域，为视觉研究者提供了一种按样本自适应分配视觉计算预算、而非固定比例压缩的 VLM 推理范式。
原文依据 enables adaptive visual token acquisition through a coarse-to-fine approach
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在视觉问答任务上取得了显著成功，但其对大量视觉词元的依赖带来了可观的计算开销。现有高效 VLM 方法通过固定比例压缩来减少视觉词元，但它们是被动运行的，缺乏适应不同任务需求的能力。这引出了一个根本性问题：VLM 能否自主确定每个样本所需的最少视觉词元数？受人类主动视觉机制启发，我们提出 AdaptVision，一种通过由粗到细方式实现自适应视觉词元获取的高效 VLM 范式。该模型首先处理来自低分辨率图像的压缩视觉词元，并在必要时通过调用边界框工具裁剪关键区域来有选择地获取额外视觉信息。我们使用强化学习框架训练 AdaptVision，该框架谨慎地平衡准确率与效率。本方法的核心是解耦轮次策略优化（DTPO），它将学习目标解耦为两个部分：（1）工具学习，优化正确的工具使用；（2）准确率提升，精炼生成的回答以提高答案正确性。基于这一形式化，我们进一步解耦优势估计，为与各目标相关的词元分别计算优势。与原始 GRPO 相比，这一形式化使 AdaptVision 获得更有效的优化。在多个 VQA 基准上的大量实验表明，与最优的高效 VLM 方法相比，AdaptVision 在消耗显著更少视觉词元的同时取得了更优的性能。
英文原文摘要 1742 字符
Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.