错误归因：第一人称视频中的细粒度错误理解
Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos
提出细粒度错误归因任务，将错误定位到指令语义角色、不可逆时间点（PNR）及空间位置，并给出自动构建大规模错误样本的数据引擎 MisEngine 与统一注意力模型 MisFormer，可复用于第一人称视频的时序定位与视觉语言理解。
原文依据 We introduce Mistake Attribution (MATT), a new task for fine-grained understanding of human mistakes in egocentric videos.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出错误归因（Mistake Attribution, MATT），这是一项用于细粒度理解第一人称视频中人类错误的新任务。以往工作仅检测是否发生错误，而 MATT 将错误归因于指令中被违反的部分（语义角色）、视频中偏差变得不可逆的时刻（不可逆点，Point-of-No-Return, PNR），以及错误在 PNR 帧中出现的位置。我们开发了 MisEngine，一个数据引擎，可从现有具有丰富归因标注的数据集中自动构建错误样本。应用于大规模第一人称语料库后，MisEngine 生成了 EPIC-KITCHENS-M 和 Ego4D-M——两个比先前错误数据集大最多两个数量级的数据集。随后我们提出 MisFormer，一个统一的基于注意力的模型，用于跨语义、时序和空间维度的错误归因，并使用 MisEngine 监督进行训练。一项人类研究表明，我们 MisEngine 构建的错误样本具有生态效度，证实 EPIC-KITCHENS-M 和 Ego4D-M 可作为错误理解的可靠基准。在我们两个数据集和先前基准上的实验表明，MisFormer 作为单一统一模型，在视频语言理解、时序定位、手物交互和错误检测方面分别至少优于任务特定 SOTA 方法 6.66%、21.81%、18.7% 和 3.00%。项目页面：https://yayuanli.github.io/MATT/
英文原文摘要 1417 字符
We introduce Mistake Attribution (MATT), a new task for fine-grained understanding of human mistakes in egocentric videos. While prior work detects whether a mistake occurs, MATT attributes the mistake to what part of the instruction is violated (semantic role), when in the video the deviation becomes irreversible (the Point-of-No-Return, PNR), and where the mistake appears in the PNR frame. We develop MisEngine, a data engine that automatically constructs mistake samples from existing datasets with attribution-rich annotations. Applied to large egocentric corpora, MisEngine yields EPIC-KITCHENS-M and Ego4D-M---two datasets up to two orders of magnitude larger than prior mistake datasets.We then present MisFormer, a unified attention-based model for mistake attribution across semantic, temporal, and spatial dimensions, trained with MisEngine supervision. A human study demonstrates the ecological validity of our MisEngine-constructed mistake samples, confirming that EPIC-KITCHENS-M and Ego4D-M can serve as reliable benchmarks for mistake understanding. Experiments on both our datasets and prior benchmarks show that MisFormer, as a single unified model, outperforms task-specific SOTA methods by at least 6.66%, 21.81%, 18.7%, and 3.00% in video-language understanding, temporal localization, hand-object interaction, and mistake detection, respectively. Project page: https://yayuanli.github.io/MATT/