你看到我指向哪里了吗？基于手势的第一人称视频问答
Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering
将3D手部关键点编码为离散token并与视频帧交错输入MLLM，为第一人称视频中的指向意图理解提供了可复用的空间-时序上下文注入方式；同时发布的EgoPointVQA数据集可用于评测模型的手势定位与指代推理能力。
原文依据 we further propose Hand Intent Tokens (HINT), which encode tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model and interleaves them with the model input
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于用户的指向手势来理解和回答问题，对于下一代第一人称AI助手至关重要。然而，当前的多模态大语言模型（MLLMs）在此类任务上表现不佳，原因在于缺乏手势丰富的数据，以及它们从第一人称视频中推断细粒度指向意图的能力有限。为解决这一问题，我们提出了EgoPointVQA，一个用于基于手势的第一人称问答的数据集与基准，包含4000个合成视频和400个真实世界视频，覆盖多种指示性推理任务。在此基础上，我们进一步提出了手部意图令牌（HINT），它利用现成的重建模型从3D手部关键点生成令牌，并将这些令牌与模型输入交错，为解释指向意图提供明确的空间和时序上下文。我们表明，我们的模型在不同骨干网络和模型规模下均优于其他模型。特别是，HINT-14B在6个任务上的平均准确率达到68.1%，比最先进的InternVL3-14B高出6.6%。为进一步促进开放研究，我们将发布代码、模型和数据集。
英文原文摘要 1125 字符
Understanding and answering questions based on a user's pointing gesture is essential for next-generation egocentric AI assistants. However, current Multimodal Large Language Models (MLLMs) struggle with such tasks due to the lack of gesture-rich data and their limited ability to infer fine-grained pointing intent from egocentric video.To address this, we introduce EgoPointVQA, a dataset and benchmark for gesture-grounded egocentric question answering, comprising 4000 synthetic and 400 real-world videos across multiple deictic reasoning tasks.Built upon it, we further propose Hand Intent Tokens (HINT), which encode tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model and interleaves them with the model input to provide explicit spatial and temporal context for interpreting pointing intent.We show that our model outperforms others in different backbones and model sizes.In particular, HINT-14Bachieves 68.1% accuracy, on average over 6 tasks, surpassing the state-of-the-art, InternVL3-14B, by 6.6%.To further facilitate the open research, we will release the code, model, and dataset.