看、想、做：通过识别开关状态教多模态智能体有效交互图形用户界面
See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles
把「当前状态感知」显式引入多模态智能体的推理链，对做 GUI 视觉定位与指令遵循的研究者提供了一个可复用的两阶段范式：先读当前视觉状态，再与指令目标状态比对后决策，可迁移到任何需要状态判断的视觉交互任务。
原文依据 we propose State-aware Reasoning (StaR), a multimodal reasoning method that enables agents to perceive the current toggle state
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态智能体的出现促进了图形用户界面（GUI）中的有效交互，尤其是在无处不在的 GUI 控制场景中。然而，它们无法可靠执行开关控制指令，这仍是一个关键瓶颈。为研究这一问题，我们基于公开数据集构建了一个包含二值开关指令的状态控制基准。现有智能体的评测结果表明其可靠性明显不足，尤其是当当前开关状态已与目标状态一致时。为应对该挑战，我们提出状态感知推理（State-aware Reasoning, StaR），一种多模态推理方法，使智能体能够感知当前开关状态、从指令中推断目标状态并据此行动。在四个多模态智能体上的实验表明，StaR 可将开关指令执行准确率提升 30% 以上。在三个公开智能体基准上的进一步评测显示，StaR 也能提升通用智能体任务性能。最后，在动态环境中的评测凸显了 StaR 在实际应用中的潜力。代码与基准：https://github.com/ZrW00/StaR。
英文原文摘要 1141 字符
The advent of multimodal agents facilitates effective interaction within graphical user interface (GUI), especially in ubiquitous GUI control. However, their inability to reliably execute toggle control instructions remains a key bottleneck. To investigate this, we construct a state control benchmark with binary toggle instructions derived from public datasets. Evaluation results of existing agents demonstrate their notable unreliability, particularly when the current toggle state already matches the desired state. To address the challenge, we propose State-aware Reasoning (StaR), a multimodal reasoning method that enables agents to perceive the current toggle state, infer the desired state from the instruction, and act accordingly. Experiments on four multimodal agents demonstrate that StaR can improve toggle instruction execution accuracy by over 30%. Further evaluations on three public agentic benchmarks show that StaR also enhances general agentic task performance. Finally, evaluations on a dynamic environment highlight the potential of StaR for real-world applications. Code and benchmark: https://github.com/ZrW00/StaR.