揭示与评估 GUI 视觉定位中的幻觉
Exposing and Evaluating Hallucinations for GUI Grounding
把 GUI 视觉定位中的幻觉拆成「混淆幻觉」与「捏造幻觉」两类，并给出可复用的评测子集设计（解析子集 + 幻觉子集），可直接迁移到其他视觉定位任务上做失效模式诊断。
原文依据 we expose two major types of hallucinations in GUI grounding: 1) Confusion Hallucination
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有的 GUI 基准主要关注评估模型的综合能力，却在很大程度上忽视了视觉定位任务中的幻觉现象，而这一现象对 GUI 理解的可靠性至关重要。在本工作中，我们揭示了 GUI 视觉定位中的两类主要幻觉：1）混淆幻觉，即错误地选中了干扰元素；2）捏造幻觉，即用看似合理的坐标幻觉出并不存在的元素。为系统性地探究其成因，我们提出了 GUI-HalluBench，该基准包含两个互补的子集：一个解析子集，用于评估 GUI 元素的结构化表示；一个幻觉子集，用于衡量模型在困难条件下的视觉定位鲁棒性。这一设计使我们能够将幻觉模式与前置能力的缺陷关联起来：解析错误与捏造幻觉和混淆幻觉都密切相关。对最优模型的实验证实了这些关联，为幻觉的根本成因提供了新的洞见，并指导更可靠的 GUI 理解工具的研发。
英文原文摘要 1142 字符
Existing GUI benchmarks primarily focus on evaluating models' comprehensive capabilities but largely overlook hallucination phenomena in grounding tasks, which are crucial to the reliability of GUI understanding. In this work, we expose two major types of hallucinations in GUI grounding: 1) Confusion Hallucination, where distractor elements are mistakenly selected, and 2) Fabricated Hallucination, where nonexistent elements are hallucinated with plausible coordinates. To systematically investigate their origins, we introduce GUI-HalluBench, a benchmark comprising two complementary subsets: a parsing subset for assessing structural representation of GUI elements and a hallucination subset for measuring grounding robustness under challenging conditions. This design allows us to associate hallucination patterns with deficiencies in prerequisite abilities: parsing errors are closely tied to both fabricated and confusion hallucinations. Experiments on state-of-the-art models confirm these connections, offering new insights into the root causes of hallucinations and guiding the development of more reliable GUI understanding tools.