AXG-Reasoner：基于视觉语言模型的长任务视频错误检测与解释
AXG-Reasoner: Error Detection and Explanation in Long Task Videos with Vision-Language Models
用冻结 VLM 配合动作执行图（AXG）与时序动作分割模型，把长视频切分成细粒度子动作片段，再用少量关键帧和增强提示逐段查询 VLM 做错误检测与解释；这套「TAS 分段 + 图对齐 + 稀疏关键帧查询」的流程可直接迁移到长视频中稀疏事件定位与细粒度时序推理任务。
原文依据 we decompose each action segment of the video, obtained by TAS, into a sequence of fine-grained subactions by aligning it with the AXG
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论虚拟任务助手必须识别并解释用户的错误，才能提供有效的纠正性指导。本文研究长任务视频中的错误推理问题，即检测并解释错误。尽管近期的视觉语言模型（VLM）在视觉问答方面展现出强大能力，但它们难以关注到长任务视频中与错误相关的稀疏时空线索。我们提出一种错误推理框架 AXG-Reasoner，它利用冻结的 VLM，并结合所提出的动作执行图（Action eXecution Graph，AXG）以及一个从正常（无错误）视频中获得并训练的时序动作分割（TAS）模型。为使 VLM 能够关注到与错误相关的稀疏时空线索，我们将 TAS 得到的每个动作片段与 AXG 对齐，从而分解为一系列细粒度子动作。对于每个子动作片段，我们使用少量关键帧和增强提示查询 VLM，以检测并解释错误，从而实现数据高效。为避免昂贵的人工子动作标注，我们开发了一种利用基础模型从训练视频中自动构建 AXG 的方法。在 EgoPER 和 CaptainCook4D 上的大量实验表明，我们的方法通过有效识别时空线索，在错误解释上持续优于 VLM 基线，并在错误检测上达到最优性能。
英文原文摘要 1446 字符
Virtual task assistants must recognize and explain users' mistakes to provide effective and corrective guidance. In this paper, we address the problem of error reasoning in long task videos, which is to detect and explain errors. Although recent Vision-Language Models (VLMs) demonstrate strong capabilities in visual question answering, they struggle to attend to the sparse spatiotemporal cues associated with errors in long task videos. We introduce an error reasoning framework, AXG-Reasoner, that leverages a frozen VLM in conjunction with a proposed Action eXecution Graph (AXG) and a temporal action segmentation (TAS) model, obtained and learned from normal (error-free) videos. To enable VLMs to attend to the sparse spatiotemporal cues associated with errors, we decompose each action segment of the video, obtained by TAS, into a sequence of fine-grained subactions by aligning it with the AXG. For each subaction segment, we query the VLM using a small number of keyframes and enhanced prompts to detect and explain errors, enabling data efficiency. To avoid costly manual subaction annotations, we develop a method to automatically construct AXG from training videos using foundation models. Extensive experiments on EgoPER and CaptainCook4D show that our method consistently improves over VLM baselines in error explanation by effectively identifying spatiotemporal cues and achieves state-of-the-art performance in error detection.