Graph2Eval：通过知识图谱为智能体自动生成多模态任务
Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs
为多模态智能体提供可自动扩展的评测任务生成管线：用知识图谱作为结构化任务空间，通过子图采样与元路径构造文档理解与网页交互任务，可直接用于构建视觉-语言智能体的评测基准，缓解静态数据集规模受限的问题。
原文依据 Graph2Eval, a knowledge-graph-driven framework for automated, scalable, and semantically grounded agent task generation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论随着多模态大语言模型驱动的智能体在自主性与泛化能力上的不断进步，传统静态数据集面临固有的可扩展性局限，不足以全面评估其在日益复杂多样任务上的能力。已有研究尝试利用大语言模型生成智能体任务，但由于大语言模型固有的幻觉问题以及缺乏对内部数据关系的建模，这些任务往往存在语义不一致与可解性问题。为应对这些挑战，本文提出 Graph2Eval，一个由知识图谱驱动的框架，用于自动化、可扩展且语义有据的智能体任务生成。其核心在于，Graph2Eval 利用由异构外部数据源构建的知识图谱作为结构化任务空间，通过子图采样以及由任务模板与元路径策略引导的任务构造来生成多模态智能体任务。为进一步保证任务可靠性，一个基于节点可达性分析、大语言模型评分与相似度分析的多阶段过滤流程确保了所生成任务的多样性与可解性。通过统一 RAG 智能体与 Web 智能体两类场景，Graph2Eval 能够高效生成多模态文档理解任务与多步网页交互任务。我们以 Graph2Eval-Bench 实例化该框架，这是一个精心构建的数据集，包含 1,319 个任务，覆盖文档理解与网页交互场景。大量实验表明，Graph2Eval 相较基线平均将任务语义一致性提升 20%、可解性提升 17%，同时 Graph2Eval-Bench 能有效区分智能体性能，为自动化智能体评估提供了新视角。
英文原文摘要 1695 字符
As multimodal LLM-driven agents advance in autonomy and generalization, traditional static datasets face inherent scalability limitations and are insufficient for fully assessing their capabilities in increasingly complex and diverse tasks. Existing studies have attempted to generate agent tasks using LLMs, but due to the inherent hallucinations of LLMs and the lack of internal data relationship modeling, these tasks often exhibit semantic inconsistencies and solvability issues. To address these challenges, we introduce Graph2Eval, a knowledge-graph-driven framework for automated, scalable, and semantically grounded agent task generation. At its core, Graph2Eval leverages a knowledge graph built from heterogeneous external data sources as a structured task space, generating multimodal agent tasks through subgraph sampling and task construction guided by task templates and meta-path strategies. To further ensure task reliability, a multi-stage filtering pipeline based on node reachability analysis, LLM scoring, and similarity analysis ensures the diversity and solvability of the generated tasks. By unifying both RAG Agent and Web Agent scenarios, Graph2Eval enables efficient generation of multimodal document understanding tasks and multi-step web interaction tasks. We instantiate the framework with Graph2Eval-Bench, a curated dataset of 1,319 tasks spanning document understanding and web interaction scenarios. Extensive experiments show that, on average, Graph2Eval improves task semantic consistency by 20% and solvability by 17% over baselines, while Graph2Eval-Bench effectively distinguishes agent performance, offering a new perspective on automated agent evaluation.