视觉-语言对齐辅助的原型动作推理用于第一视角动作预测
Prototypical Action Reasoning Facilitated by Vision-Language Alignment for Egocentric Action Anticipation
将无约束的时序预测转化为由视觉-语言模型学到的解耦动词/名词原型引导的条件预测，这一「语义锚点+原型检索」思路可迁移到其他高不确定性的时序预测任务（如轨迹预测、事件预测）。
原文依据 transforming the unconstrained temporal prediction problem into a conditional forecasting task guided by well-defined semantic concepts
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论第一视角动作预测旨在从视频中推断未来动作，这对具身智能系统至关重要。然而，未来的内在随机性带来了显著的预测不确定性，阻碍了该方向的发展。现有方法通常采用端到端方式建模整体时空上下文，但往往缺乏显式的语义推理能力，难以应对开放式的未来不确定性。为解决这些挑战，我们提出了一种由视觉-语言对齐辅助的原型动作推理框架（PAR-VLA），它利用视觉-语言模型的语义对齐能力，为动词和名词学习解耦的视觉原型。这些原型作为稳健的语义锚点，将无约束的时序预测问题转化为由明确定义的语义概念引导的条件预测任务。我们的多阶段框架首先从视觉-语言模型中提取具有视觉基础且与文本对齐的原型组，为每个类别学习多个原型以捕捉类内多样性。随后，一种新颖的原型上下文推理引导的动词-名词编码分支根据视觉观察动态检索最相关的动词和名词概念，并显式建模它们的交互以引导时序预测。此外，我们引入了双流共生预测解码器，以在预测过程中更精细地捕捉动词与名词之间的相互依赖关系。实验结果表明，PAR-VLA 达到了最优性能，并在应对未来不确定性方面展现出强大能力。
英文原文摘要 1686 字符
Egocentric Action Anticipation aims to infer future actions from videos, which is crucial for embodied AI systems. However, its advancement is hindered by the inherent stochasticity of the future, which introduces significant prediction uncertainty. Prevailing methods typically adopt an end-to-end approach to model holistic spatiotemporal contexts, yet they often lack explicit semantic reasoning capabilities, making it difficult to handle open-ended future uncertainties. To address these challenges, we propose a Prototypical Action Reasoning Framework Facilitated by Vision-Language Alignment (PAR-VLA), which leverages the semantic alignment capability of vision-language models to learn disentangled visual prototypes for verbs and nouns. These prototypes serve as robust semantic anchors, transforming the unconstrained temporal prediction problem into a conditional forecasting task guided by well-defined semantic concepts. Our multi-stage framework first extracts visually-grounded and text-aligned prototype groups from a VLM, learning multiple prototypes per category to capture intra-class diversity. Subsequently, a novel Prototypical Context Reasoning-guided Verb-Noun Encoding branch dynamically retrieves the most relevant verb and noun concepts based on visual observations and explicitly models their interactions to guide temporal anticipation. Furthermore, we introduce Dual-Stream Symbiotic Predictive Decoders to more finely capture the interdependencies between verbs and nouns during the prediction process. Experiments Results demonstrate that PAR-VLA achieves state-of-the-art performance and exhibits a strong capability in dealing with future uncertainty.