MoVie：利用人体运动拓宽动作检测的视野
MoVie: Broaden Your Views with Human Motion for Action Detection
将人体骨架序列作为运动先验，通过可学习运动字典分解和正交变换正则化，引导RGB视频特征学习，为视频动作检测提供了一种融合异构模态的通用思路。
原文依据 we propose MoVie, a unified Motion-Video processing framework that uses structured human motion as a bridge between the two signals.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频中的人体动作检测既需要语义识别，也需要对运动进行准确建模。尽管近期的视频基础模型提升了视觉语义理解，但由于运动表征能力有限，它们仍难以捕捉复杂和组合性动作。人体骨架序列明确描述了身体结构和运动，提供了有价值的物理和几何运动信息，能够补充RGB视频。然而，融合视频和骨架模态面临两个关键挑战：(i) 标签驱动的骨架特征过于粗糙，难以描述细粒度运动；(ii) 骨架运动与RGB视频处于异构特征空间，当前的融合策略常导致特征干扰。为解决这些问题，我们提出MoVie，一个统一的运动-视频处理框架，利用结构化人体运动作为两种信号之间的桥梁。我们首先提出结构运动投影模块，使用可学习运动字典将运动分解为基元组件，以生成细粒度描述符。然后，我们设计运动引导的特征正则化机制，通过基于正交性的变换将视觉特征与运动对齐，使细粒度运动线索能够引导视觉表征而不破坏语义多样性。在Toyota Smarthome Untrimmed、Charades、Multi-THUMOS和PKU-MMD数据集上的大量评估表明，MoVie显著提升了最优动作检测性能。
英文原文摘要 1521 字符
Human action detection in videos requires both semantic recognition and accurate modeling of motion. While recent video foundation models have advanced visual semantics, they still struggle to capture complex and compositional actions due to the limited representation ability of motion. Human skeleton sequences, which explicitly describe the body structure and movement, provide valuable physical and geometric motions that complement RGB videos. However, combining video and skeleton modalities faces two key challenges: (i) label-driven skeleton features are too coarse to describe fine-grained motion, and (ii) skeleton motion and RGB video lie in heterogeneous feature spaces, so current fusion strategies often cause feature interference. To address these, we propose MoVie, a unified Motion-Video processing framework that uses structured human motion as a bridge between the two signals. We first propose a Structural Motion Projection module that decomposes motion into primitive components using a learnable motion dictionary, to produce fine-grained descriptors. Then, we design a Motion-guided Feature Regularization mechanism that aligns visual features with motion through an orthogonality-based transformation, so that fine-grained motion cues can guide visual representations without collapsing semantic diversity. Extensive evaluations on Toyota Smarthome Untrimmed, Charades, Multi-THUMOS and PKU-MMD datasets demonstrate that MoVie significantly improves state-of-the-art action detection performance.