Demo2Tutorial：从人类经验到多模态软件教程
Demo2Tutorial: From Human Experience to Multimodal Software Tutorials
论文将屏幕录制与交互日志转化为结构化图文教程，其多模态动作解析与步骤规划思路可迁移到视觉-语言模型中的指令生成与GUI智能体训练，为利用无标注人类交互数据提供了一种可复用的表征蒸馏方法。
原文依据 transforms this experience captured via screen recordings and interaction logs into structured, multimodal software tutorials
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论数字环境中的人类经验提供了大量真实、未经剪辑的交互资源，其中蕴含丰富的程序性知识，但目前尚未被充分探索。我们提出 Demo2Tutorial，一个将屏幕录制和交互日志捕获的经验转化为结构化、多模态软件教程的框架，用于教学人类和智能体。Demo2Tutorial 首先通过专用记录器收集人类经验，然后使用多模态动作解析器解析原始经验，以重建感知、动作和意图。接着，步骤规划器将这些步骤抽象为表示目标和步骤的分层任务图。最后，教程合成器将解析后的经验转化为结构化、可复用的图文指令。我们在基于官方软件文档构建的新基准上评估教程生成质量。我们进一步证明，这种蒸馏表示有益于（i）人类学习，通过自动生成多模态教程，以及（ii）智能体学习，通过改进下游 GUI 智能体的规划和泛化能力。实验表明，Demo2Tutorial 生成的教程质量高，超越人类编写的教程，并显著优于基线方法，同时实现更快的人类任务完成和更好的 GUI 智能体规划，证明从人类经验中蒸馏的结构化教程可以作为有效的知识表示，推动人类学习和智能体能力的发展。代码和数据将在 https://github.com/showlab/Demo2Tutorial 提供。
英文原文摘要 1572 字符
Human experience in digital environments offers a vast, underexplored resource of authentic, untrimmed interactions that contain rich procedural knowledge. We introduce Demo2Tutorial, a framework that transforms this experience captured via screen recordings and interaction logs into structured, multimodal software tutorials for teaching both humans and agents. Demo2Tutorial first collects human experience via a dedicated recorder, then parses raw experience using a multimodal Action Parser to reconstruct perception, action, and intent. A Step Planner then abstracts these steps into hierarchical task graphs representing goals and steps. Finally, a Tutorial Composer transforms the parsed experience into structured, reusable image-text instructions. We evaluate the tutorial generation quality on a new benchmark derived from official software documentation. We further demonstrate that this distilled representation benefits (i) human learning, by automatically generating multimodal tutorials, and (ii) agent learning, by improving downstream GUI-agent planning and generalization. Experiments show Demo2Tutorial produces high-quality tutorials that surpass human-authored ones and significantly outperform baseline methods, while enabling both faster human task completion and improved GUI agent planning, demonstrating that structured tutorials distilled from human experience can serve as effective knowledge representations for advancing both human learning and agent capabilities. Code and data will be available at: https://github.com/showlab/Demo2Tutorial