GUIDE：面向开放式 GUI 任务中理解与辅助用户的基准
GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks
为视觉研究者提供了一个屏幕录制序列上的行为状态识别与意图预测基准，可用于训练和评测多模态大模型对连续界面操作序列的时序理解与用户建模能力，并验证了结构化用户上下文对辅助决策的显著增益。
原文依据 GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论图形用户界面（GUI）智能体有潜力协助用户与复杂软件（如 PowerPoint、Photoshop）进行交互。既有研究主要聚焦于通过点击和按键来自动化用户操作，但这一范式忽视了人的意图——用户更看重在保持自主性的同时探索、迭代与完善自己想法的能力。为从自动化走向协作，GUI 智能体必须理解用户正在做什么以及为什么这样做。我们提出 GUIDE（GUI 用户意图检测评测），一个评测 AI 模型在开放式 GUI 任务中感知用户行为、推断意图并提供协助能力的基准。GUIDE 包含 67.5 小时的屏幕录制，来自 120 名新手用户的演示，配有出声思维叙述，覆盖 10 款软件。GUIDE 定义了三个任务——(i) 行为状态检测、(ii) 意图预测、(iii) 帮助预测，用于测试模型识别行为状态、推理目标以及判断何时与如何提供帮助的能力。对八个最优多模态模型的评测显示，所有模型均表现不佳，在行为状态预测与帮助预测上仅达到 44.6% 和 55.0% 的准确率。然而，提供用户上下文显著提升了性能，使帮助预测最高提升 50.2 个百分点，凸显了结构化用户理解在有效协助中的关键作用。我们的数据集见 https://guide-bench.github.io。
英文原文摘要 1479 字符
Graphical User Interface (GUI) agents have the potential to assist users in interacting with complex software (e.g., PowerPoint, Photoshop). While prior research has primarily focused on automating user actions through clicks and keystrokes, this paradigm overlooks human intention, where users value the ability to explore, iterate, and refine their ideas while maintaining agency. To move beyond automation and toward collaboration, GUI agents must understand what users are doing and why. We introduce GUIDE (GUI User Intent Detection Evaluation), a benchmark that evaluates AI models on their ability to perceive user behavior, infer intent, and provide assistance in open-ended GUI tasks. GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations, across 10 software. GUIDE defines three tasks--(i) Behavior State Detection, (ii) Intent Prediction, and (iii) Help Prediction that test a model's ability to recognize behavior state, reason about goals, and decide when and how to help. Evaluations across eight state-of-the-art multimodal models reveal that all models struggled, achieving only 44.6% and 55.0% accuracy on behavior state and help prediction. However, providing user context significantly improved the performance, raising help prediction by up to 50.2pp, highlighting the critical role of structured user understanding in effective assistance. Our dataset is available at https://guide-bench.github.io.