HATS：面向 GUI 智能体的难度感知轨迹合成
HATS: Hardness-Aware Trajectory Synthesis for GUI Agents
对做 VLM 智能体数据的人来说，这篇给出了一个可复用的思路：用「语义歧义程度」作为难度信号来驱动轨迹采集，并用指令-执行一致性校验做闭环修复，可直接迁移到其他视觉-语言序列决策任务的数据构造中。
原文依据 We define hardness as the degree of semantic ambiguity associated with an action
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论由大型视觉语言模型（VLM）驱动的图形用户界面（GUI）智能体在自动化数字任务方面已展现出显著潜力，这也凸显了高质量轨迹数据对有效训练智能体的必要性。然而，现有的轨迹合成流程往往训练出无法泛化到简单交互之外的智能体。我们将这一局限归因于对语义歧义动作的忽视——这类交互的含义依赖于上下文、依赖于序列顺序或在视觉上存在歧义。此类动作对真实世界的鲁棒性至关重要，但在当前数据集中占比不足且处理不佳，导致任务指令与执行之间出现语义错位。为解决这些问题，我们提出 HATS，一个难度感知的轨迹合成框架，旨在缓解语义歧义带来的影响。我们将难度定义为与某个动作相关的语义歧义程度，并设计了两个互补模块：（1）难度驱动的探索，引导数据采集朝向歧义但信息量丰富的交互；（2）对齐引导的精炼，迭代地验证并修复指令与执行的一致性。两个模块以闭环方式运行——探索为精炼提供具有挑战性的轨迹，而精炼的反馈又更新难度信号以指导未来的探索。大量实验表明，使用 HATS 训练的智能体在基准 GUI 环境中持续优于最优基线。
英文原文摘要 1556 字符
Graphical user interface (GUI) agents powered by large vision-language models (VLMs) have shown remarkable potential in automating digital tasks, highlighting the need for high-quality trajectory data to support effective agent training. Yet existing trajectory synthesis pipelines often yield agents that fail to generalize beyond simple interactions. We identify this limitation as stemming from the neglect of semantic-ambiguous actions--interactions whose meanings are context-dependent, sequentially dependent, or visually ambiguous. Such actions are crucial for real-world robustness but are under-represented and poorly processed in current datasets, leading to semantic misalignment between task instructions and execution. To address these issues, we propose HATS, a Hardness-Aware Trajectory Synthesis framework designed to mitigate the impact of semantic ambiguity. We define hardness as the degree of semantic ambiguity associated with an action and develop two complementary modules: (1) a hardness-driven exploration that guides data collection toward ambiguous yet informative interactions, and (2) an alignment-guided refinement that iteratively validates and repairs instruction-execution consistency. The two modules operate in a closed-loop manner--exploration supplies refinement with challenging trajectories, while refinement feedback updates the hardness signal to guide future exploration. Extensive experiments show that agents trained with HATS consistently outperform state-of-the-art baselines across benchmark GUI environments.