FocusUI：通过位置保持的视觉词元选择实现高效 UI 视觉定位
FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection
针对高分辨率截图词元冗余问题，提出指令条件化 patch 级监督与 PosPad 位置连续性保持的视觉词元选择方案，可直接迁移到任意高分辨率 VLM 的视觉编码压缩与定位任务中。
原文依据 we introduce a PosPad strategy, which compresses each contiguous sequence of dropped visual tokens into a single special marker
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在处理用户界面（UI）视觉定位任务上表现强劲，这得益于其处理越来越高分辨率截图的能力。然而，截图会被词元化为数千个视觉词元（例如 2K 分辨率下约 4,700 个），这带来显著的计算开销并稀释注意力。相比之下，人类在与 UI 交互时通常只关注感兴趣区域。在本工作中，我们开创了高效 UI 视觉定位这一任务。我们提出 FocusUI，一个高效的 UI 视觉定位框架，它选择与指令最相关的图像块，同时保持位置连续性以实现精确定位。FocusUI 解决两个关键挑战：（1）通过构建图像块级监督来消除视觉编码中的冗余词元，该监督将指令条件化得分与基于规则的 UI 图得分融合，降低大面积同质区域的权重，从而选出彼此区分且与指令相关的视觉词元；（2）在视觉词元选择过程中保持位置连续性。我们发现，通用的视觉词元剪枝方法在 UI 视觉定位任务上会因位置信息被破坏而出现严重精度下降。为此，我们引入 PosPad 策略，将每一段连续的被丢弃视觉词元压缩为一个特殊标记，置于该序列的最后一个索引处，以保持位置连续性。在四个视觉定位基准上的全面实验表明，FocusUI 超越了针对 GUI 的基线方法。在 ScreenSpot-Pro 基准上，FocusUI-7B 相比 GUI-Actor-7B 取得 3.7% 的性能提升。即使仅保留 30% 的视觉词元，FocusUI-7B 的性能也仅下降 3.2%，同时实现最高 1.44 倍的推理加速和 17% 的峰值 GPU 显存降低。
英文原文摘要 1804 字符
Vision-Language Models (VLMs) have shown strong performance on User Interface (UI) grounding tasks, driven by their ability to process increasingly high-resolution screenshots. However, screenshots are tokenized into thousands of visual tokens (e.g., about 4,700 for 2K resolution), which incurs significant computational overhead and dilutes attention. In contrast, humans typically focus on regions of interest when interacting with UIs. In this work, we pioneer the task of efficient UI grounding. We propose FocusUI, an efficient UI grounding framework that selects patches most relevant to the instruction while preserving positional continuity for precise grounding. FocusUI addresses two key challenges: (1) eliminating redundant tokens in visual encoding by constructing patch-level supervision that fuses an instruction-conditioned score with a rule-based UI-graph score, down-weighting large homogeneous regions to select distinct and instruction-relevant visual tokens; and (2) preserving positional continuity during visual token selection. We find that general visual token pruning methods suffer from severe accuracy degradation on UI grounding tasks due to broken positional information. To address this, we introduce a PosPad strategy, which compresses each contiguous sequence of dropped visual tokens into a single special marker placed at the sequence's last index to preserve positional continuity. Comprehensive experiments on four grounding benchmarks demonstrate that FocusUI surpasses GUI-specific baselines. On the ScreenSpot-Pro benchmark, FocusUI-7B achieves a 3.7% performance improvement over GUI-Actor-7B. Even with only 30% visual token retention, the performance of FocusUI-7B drops by just 3.2%, while achieving up to 1.44x faster inference and 17% lower peak GPU memory.