从极性看运动:面向事件相机动作识别的极性知识增强框架
Seeing Motion Through Polarity for Event-based Action Recognition
把事件相机的正负极性显式解耦为运动线索,并用大语言模型对极性诱导的运动动态做语义推理,再与视觉特征对齐;这套『极性分支+语言推理+跨模态对齐』的做法可迁移到任何事件流或高动态场景的时序理解任务。
原文依据 PMC decouples positive and negative polarities to capture polarity-sensitive motion cues
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于事件的动作识别(EAR)为在挑战性条件下理解动态行为提供了一条有前景的途径。近期视觉语言模型的进展将跨模态学习范式引入 EAR,使模型能够将事件流与文本语义关联起来,从而增强概念理解。然而,现有方法通常忽略了事件数据所固有的、由极性驱动的运动线索,导致时空表征欠佳。为解决这一局限,我们提出了极性知识增强框架(POKER),该框架在视觉与文本模态上显式地融入事件极性感知的运动知识。POKER 由两个协同组件构成:极性运动捕获器(PMC)与极性运动推理器(PMR)。具体而言,PMC 解耦正极性与负极性以捕获极性敏感的运动线索,而 PMR 则通过大语言模型对极性诱导的运动动态进行语义分析。通过极性对齐,POKER 将语义推理与视觉动态耦合起来,从而获得更具判别力的表征。在多个基准上的大量实验表明,POKER 在多种事件表征下均能提升性能。
英文原文摘要 1276 字符
Event-based Action Recognition (EAR) provides a promising pathway for understanding dynamic behaviors under challenging conditions. Recent progress in vision-language models has introduced a cross-modal learning paradigm into EAR, enabling models to associate event streams with textual semantics for enhancing conceptual understanding. However, existing methods typically overlook the intrinsic polarity-driven motion cues that are fundamental to event data, leading to suboptimal spatiotemporal representations. To address this limitation, we propose a POlarity Knowledge Enhanced framework (POKER), which explicitly incorporates event polarity-aware motion knowledge across visual and textual modalities.POKER consists of two synergistic components: Polarity Motion Capturer (PMC) and Polarity Motion Reasoner (PMR). Specifically, PMC decouples positive and negative polarities to capture polarity-sensitive motion cues, while PMR semantically analyzes polarity-induced motion dynamics via large language models. Through the polarity alignment, POKER couples semantic reasoning with visual dynamics, achieving more discriminative representations. Extensive experiments on multiple benchmarks demonstrate that POKER enhances performance across diverse event representations.