局部运动至关重要：一种用于从视频中进行强化学习预训练的分解-重组范式
Local Motion Matters: A Deconstruct-Recompose Paradigm for Reinforcement Learning Pre-training from Videos
从视频中通过跟踪局部点（原子动作）学习可迁移的运动表征，并用潜在动力学模型聚合，为视觉驱动的机器人策略学习提供预训练方案；其双注意力编码器与运动聚合词元设计可复用于视频运动理解与模仿学习。
原文依据 we propose a novel Deconstruct-Recompose Paradigm (DRP) for learning transferable local motion representations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于大规模视频的预训练来提升强化学习效率具有前景，但仍面临挑战。现有方法通常将智能体视为不可分割的实体，全局地建模运动模式。这种全局建模与形态紧密耦合，阻碍了跨域迁移。相比之下，尽管全局运动差异巨大，不同智能体的局部组件却表现出相似的运动模式。基于这一洞察，我们提出了一种新颖的分解-重组范式（DRP），用于学习可迁移的局部运动表征。具体而言，在分解阶段，我们识别多个局部点并跟踪它们的逐帧运动，将每个点定义为一个原子动作。我们引入双注意力编码器（DAE）从这些原子动作中学习局部运动表征，捕捉其时空关系。在重组阶段，我们通过潜在动力学模型学习，使用可学习的运动聚合词元‘[MAT]’来组合局部运动表征。此外，一个适配器桥接局部运动与下游动作特定动力学，以加速策略学习。大量实验表明，我们的方法能有效迁移到多样的机器人控制与操作任务，显著提升样本效率与性能。
英文原文摘要 1341 字符
Pre-training on large-scale videos to improve reinforcement learning efficiency is promising yet remains challenging. Existing methods typically treat the agent as an indivisible entity, modeling motion patterns globally. Such global modeling is tightly coupled with the morphology, hindering transfer across domains. In contrast, despite the vast disparity in global motions, the local components exhibit similar motion patterns across different agents. Building on this insight, we propose a novel Deconstruct-Recompose Paradigm (DRP) for learning transferable local motion representations. Specifically, in the Deconstruct phase, we identify multiple local points and track their frame-wise motions, defining each as an Atomic Action. We introduce a Dual-Attention Encoder (DAE) to learn local motion representations from these Atomic Actions, capturing their spatiotemporal relationships. In the Recompose phase, we compose local motion representations with a learnable Motion Aggregation Token '[MAT]' via latent dynamics model learning. Additionally, an adapter bridges local motion and downstream action-specific dynamics to accelerate policy learning. Extensive experiments demonstrate that our method effectively transfers to diverse robotic control and manipulation tasks, significantly improving sample efficiency and performance.