UI-Lens：评估通用多模态大语言模型自动化UI显示质量保证的潜力
UI-Lens: Assessing General MLLMs' Potential to Automate UI Display Quality Assurance
该基准揭示了MLLM/VLM在细粒度元素边界理解和序列界面语义一致性上的严重不足，为视觉研究提供了可复用的评测协议和失败模式分析，可推动面向复杂真实界面的细粒度感知与推理方法研究。
原文依据 we present UI-Lens, the first multi-dimensional UI display detection benchmark for Chinese-language UI scenarios.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论用户界面（UI）显示缺陷检测带来的挑战远超UI理解，需要细粒度的元素边界理解、缺失内容检测以及对序列界面语义一致性的推理。然而，多模态大语言模型（MLLMs）和视觉语言模型（VLMs）在真实复杂界面中检测UI缺陷的能力尚未得到系统验证。为填补这一空白，我们提出了UI-Lens，这是首个面向中文UI场景的多维度UI显示检测基准。该数据集包含由设计专家精心标注的4,759个页面，覆盖六个核心显示缺陷类别。我们对10个主流模型（8个闭源、2个开源）进行了系统评估。结果表明当前模型存在明显不足：对于需要细粒度元素边界理解的任务，性能接近随机水平，在文本溢出和容器重叠上的任务平均F1分数分别为20.36%和31.21%；对于序列界面语义一致性（如文本不一致），任务平均F1分数仅为10.61%，表明性能严重不足。我们发布UI-Lens以促进面向真实复杂界面中具有细粒度边界感知的更鲁棒UI显示缺陷检测研究。
英文原文摘要 1340 字符
User Interface (UI) display defect detection poses challenges far beyond UI understanding, requiring fine-grained element boundary understanding, missing-content detection, and reasoning about sequential interface semantic consistency. However, the capabilities of multimodal large language models (MLLMs) and vision-language models (VLMs) for detecting UI defects in realistic, complex interfaces have not been systematically validated. To fill this gap, we present UI-Lens, the first multi-dimensional UI display detection benchmark for Chinese-language UI scenarios. The dataset comprises 4,759 pages meticulously annotated by design experts, covering six core display defect categories. We conduct a systematic evaluation of 10 mainstream models (8 closed-source, 2 open-source). Results show clear shortcomings in current models: for tasks requiring fine-grained element boundary understanding, performance is near random, with task-average F1 scores of 20.36% and 31.21% on Text Overflow and Container Overlap, respectively; for sequential interface semantic consistency (e.g., Text Inconsistency), the task-average F1 score is only 10.61%, indicating severe underperformance. We release UI-Lens to catalyze research toward more robust UI display defect detection with fine-grained boundary awareness in realistic, complex interfaces.