ProSoftArena：在专业软件环境中基准测试多模态智能体的分层能力
ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments
为视觉-语言-动作智能体在专业软件中的操作能力提供了首个五级能力层级和可执行评估环境，其基于执行的评估框架和人机协同效率量化方法可直接用于评测GUI智能体在真实软件工作流中的表现。
原文依据 we introduce ProSoftArena, a comprehensive benchmark and platform specifically designed for evaluating multi-modal agents in professional software environments
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态智能体在通用计算机使用任务上正快速进步。然而，现有基准仍主要局限于网页浏览器和基础应用，未能涵盖主导现实世界科学与工业实践的专业软件工作流。为弥补这一空白，我们推出 ProSoftArena，一个专为评估专业软件环境中多模态智能体而设计的综合基准与平台。我们建立了首个面向专业软件操作的五级能力层级，并策划了涵盖 6 个学科和 13 个核心专业应用的 456 个真实任务基准。为确保可靠评估，我们构建了可执行的真实计算机环境与基于执行的评估框架，并独特地引入人机协同评估范式以量化智能体的协作效率。大量实验表明，即使表现最佳的智能体在软件级任务（L2）上成功率也仅为 20.6%，在多软件工作流（L3）上则完全失败。我们的深入分析进一步提供了关于当前智能体局限性的宝贵见解，并为在专业软件环境中构建更强大智能体提出了有效设计原则。项目见：https://prosoftarena.github.io。
英文原文摘要 1351 字符
Multi-modal agents are making rapid progress on general computer-use tasks. However, existing benchmarks remain largely confined to web browsers and rudimentary applications, failing to capture the professional software workflows that dominate real-world scientific and industrial practices. To bridge this gap, we introduce ProSoftArena, a comprehensive benchmark and platform specifically designed for evaluating multi-modal agents in professional software environments. We establish the first five-level capability hierarchy for professional software manipulation, and curate a benchmark of 456 realistic tasks spanning 6 disciplines and 13 core professional applications. To ensure reliable assessment, we build an executable real-computer environment with an execution-based evaluation framework, and uniquely incorporate a human-in-the-loop evaluation paradigm to quantify agents' collaborative efficiency. Extensive experiments show that even the best-performing agent achieves only a 20.6% success rate on software-level tasks (L2) and completely fails on multi-software workflows (L3). Our in-depth analysis further provides valuable insights in current agent limitations and suggests effective design principles for building more capable agents in professional software settings. This project is available at: https://prosoftarena.github.io.