ShowUI-p：基于流的生成模型作为 GUI 灵巧手
ShowUI-p: Flow-based Generative Models as GUI Dexterous Hands
将流匹配生成模型用于 GUI 连续拖拽动作的闭环控制，为视觉智能体提供了一种从屏幕视觉观测直接生成平滑轨迹的轻量动作专家设计，可迁移到其他需要连续交互的视觉-动作任务。
原文依据 Flow-based Action Generation for drag modeling, which predicts incremental cursor adjustments from continuous visual observations via a lightweight action expert
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论构建能够进行灵巧操作的智能体，对于在机器人和数字环境中实现类人自动化至关重要。然而，现有的 GUI 智能体依赖离散的点击预测（x,y），这阻碍了需要连续、实时感知与调整的自由形式闭环轨迹（例如拖动进度条）。在本工作中，我们开发了 ShowUI-p，这是首个作为 GUI 灵巧手的基于流的生成模型，具有以下设计：（i）统一的离散-连续动作，在共享模型中整合离散点击和连续拖拽，实现跨多种交互模式的灵活适应；（ii）基于流的拖拽建模动作生成，通过轻量级动作专家从连续视觉观测中预测增量光标调整，确保轨迹平滑稳定；（iii）拖拽训练数据与基准，我们手动收集并合成了跨越五个领域（如 PowerPoint、Adobe Premiere Pro）的 20K 条拖拽轨迹，并引入了 ScreenDrag，一个具有全面在线和离线评估协议的基准，用于评估 GUI 智能体的拖拽能力。我们的实验表明，专有 GUI 智能体在 ScreenDrag 上仍然表现不佳（例如 Operator 得分为 13.27，最佳 Gemini-2.5-CUA 达到 22.18）。相比之下，ShowUI-p 仅以 450M 参数就达到了 26.98，凸显了任务的难度和方法的有效性。我们希望这项工作能推动 GUI 智能体在数字世界中实现类人灵巧控制。
英文原文摘要 1575 字符
Building intelligent agents capable of dexterous manipulation is essential for achieving human-like automation in both robotics and digital environments. However, existing GUI agents rely on discrete click predictions (x,y), which prohibits free-form, closed-loop trajectories (e.g. dragging a progress bar) that require continuous, on-the-fly perception and adjustment. In this work, we develop ShowUI-p, the first flow-based generative model as GUI dexterous hand, featuring the following designs:(i) Unified Discrete-Continuous Actions, integrating discrete clicks and continuous drags within a shared model, enabling flexible adaptation across diverse interaction modes;(ii) Flow-based Action Generation for drag modeling, which predicts incremental cursor adjustments from continuous visual observations via a lightweight action expert, ensuring smooth and stable trajectories;(iii) Drag Training data and Benchmark, where we manually collect and synthesize 20K drag trajectories across five domains (e.g. PowerPoint, Adobe Premiere Pro), and introduce ScreenDrag, a benchmark with comprehensive online and offline evaluation protocols for assessing GUI agents' drag capabilities.Our experiments show that proprietary GUI agents still struggle on ScreenDrag (e.g. Operator scores 13.27, and the best Gemini-2.5-CUA reaches 22.18). In contrast, ShowUI-p achieves 26.98 with only 450M parameters, underscoring both the difficulty of the task and the effectiveness of our approach. We hope this work advances GUI agents toward human-like dexterous control in digital world.