Minerva-Ego：面向第一人称视频理解的时空提示
Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding
把「该看哪里、何时看」的时空掩码作为提示注入前沿多模态模型，可显著提升复杂视频推理；这套提示构造与推理链标注方式可直接迁移到其他视频问答或具身感知任务的评测与增强。
原文依据 prompting frontier models with hints of 'where' and when to look yields substantial improvements in performance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频推理模型是第一人称与具身智能体的核心组件。然而，用于评估这些模型的标准基准只评价输出（例如问题的答案），而不评价中间推理步骤，且大多数基准只在文本域给出答案。我们提出 Minerva-Ego，一个用于评估复杂第一人称视觉推理的基准。我们在近期从第一人称/具身场景录制的高质量视频数据源基础上，扩展出一组具有挑战性的多步多模态问题，以及时空稠密的人工标注推理轨迹。基准实验表明，最优模型与人类表现之间仍存在较大差距。为深入探究这一差距，我们为数据集中的每条推理轨迹标注了求解问题所需的感兴趣目标，以时空掩码的形式给出。通过大量评估，我们发现，向前沿模型提示「看哪里」以及「何时看」的线索，能带来显著的性能提升。
英文原文摘要 1055 字符
Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of inter- mediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a bench- mark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatio-temporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and when to look yields substantial improvements in performance.