视觉语言模型 · CVPR2026 ·2026.06.03

面向 GUI 智能体:用于 GUI 视觉定位的视觉语言扩散模型

Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding

Shrinidhi Kumbhar、Haofu Liao、Srikar Appalaraju、Kunwar Yashraj Singh

与你有关的部分

将离散扩散视觉语言模型(如 LLaDA-V)用于 GUI 视觉定位任务,提出线性与确定性混合的掩码调度以捕捉边界框的层次几何结构,为视觉定位提供了一种非自回归的并行生成范式。

原文依据 we propose a hybrid masking schedule that combines linear and deterministic masking, improving grounding accuracy by up to 6.1 points
可迁移到 GUI 视觉定位视觉语言理解边界框预测

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

自回归(AR)视觉语言模型(VLM)长期主导多模态理解、推理和图形用户界面(GUI)视觉定位。近期,离散扩散视觉语言模型(DVLM)在多模态推理中展现出强大性能,提供双向注意力、并行词元生成和迭代细化。然而,其在 GUI 视觉定位中的潜力尚未被探索。在本工作中,我们评估离散 DVLM 能否作为 AR 模型在 GUI 视觉定位中的可行替代方案。我们将 LLaDA-V 适配于单轮动作和边界框预测,将任务框定为从多模态输入进行文本生成。为了更好地捕捉边界框几何的层次结构,我们提出了一种混合掩码调度,结合线性掩码和确定性掩码,相比使用线性掩码训练的 GUI 适配版 LLaDA-V,在步骤成功率(SSR)上提升了高达 6.1 个百分点。在涵盖 Web、桌面和移动界面的四个数据集上的评估表明,采用混合掩码的适配扩散模型始终优于线性掩码变体,并且尽管预训练有限,仍与自回归对应模型表现相当。系统性消融实验揭示,增加扩散步数、生成长度和块长度可提高准确率,但也增加延迟,且准确率在超过一定扩散步数后趋于平稳。用多样化的 GUI 领域扩展训练数据进一步将延迟降低约 1.3 秒,并在各基准上平均提升视觉定位准确率 20 个百分点。这些结果表明,离散 DVLM 是 GUI 视觉定位一个有前景的建模框架,代表了迈向基于扩散的 GUI 智能体的重要一步。

英文原文摘要 1776 字符

Autoregressive (AR) vision-language models (VLMs) have long dominated multimodal understanding, reasoning, and graphical user interface (GUI) grounding. Recently, discrete diffusion vision-language models (DVLMs) have shown strong performance in multimodal reasoning, offering bidirectional attention, parallel token generation, and iterative refinement. However, their potential for GUI grounding remains unexplored. In this work, we evaluate whether discrete DVLMs can serve as a viable alternative to AR models for GUI grounding. We adapt LLaDA-V for single-turn action and bounding-box prediction, framing the task as text generation from multimodal input. To better capture the hierarchical structure of bounding-box geometry, we propose a hybrid masking schedule that combines linear and deterministic masking, improving grounding accuracy by up to 6.1 points in Step Success Rate (SSR) over the GUI-adapted LLaDA-V trained with linear masking. Evaluations on four datasets spanning web, desktop, and mobile interfaces show that the adapted diffusion model with hybrid masking consistently outperforms the linear-masked variant and performs competitively with autoregressive counterparts despite limited pretraining. Systematic ablations reveal that increasing diffusion steps, generation length, and block length improves accuracy but also increases latency, with accuracy plateauing beyond a certain number of diffusion steps. Expanding the training data with diverse GUI domains further reduces latency by about 1.3 seconds and improves grounding accuracy by an average of 20 points across benchmarks. These results demonstrate that discrete DVLMs are a promising modeling framework for GUI grounding and represent an important step toward diffusion-based GUI agents.

标签

会议 CVPR2026
页码 27493-27502
Bibkey
来源 CVF Open Access