CoMo：从互联网视频中学习连续潜运动以实现可扩展的机器人学习
CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning
用早期时序差分与时间方向反转构造正负样本的时序对比学习，从互联网视频中无监督地提取连续潜运动表征，可直接为视觉策略学习生成伪动作标签，是一种可复用的视频表征预训练手段。
原文依据 CoMo employs an early temporal difference (Td) mechanism to increase the shortcut learning difficulty and explicitly enhance motion cues.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从互联网视频中无监督地学习潜运动对机器人学习至关重要。现有的离散方法通常通过使用小码本的向量量化来缓解因提取过多静态背景而导致的捷径学习问题。然而，这些方法存在信息损失，难以捕捉更复杂、更细粒度的动态。此外，离散潜运动的分布与连续机器人动作的分布之间存在固有差距，这阻碍了统一策略的联合学习。我们提出 CoMo，旨在从互联网规模的视频中学习更精确的连续潜运动。CoMo 采用早期时序差分（Td）机制来增加捷径学习的难度并显式增强运动线索。此外，为确保潜运动更好地捕捉有意义的前景，我们进一步提出时序对比学习（Tcl）方案。具体而言，正样本对由具有较小未来帧时序偏移的帧构成，而负样本对则通过直接反转时序方向形成。所提出的 Td 与 Tcl 协同工作，有效确保潜运动更好地聚焦于前景并强化运动线索。关键在于，CoMo 展现出强大的零样本泛化能力，使其能够为未见过的视频生成有效的伪动作标签。大量仿真与真实世界实验表明，使用 CoMo 伪动作标签联合训练的策略在扩散架构与自回归架构下均取得更优性能。代码已发布于 https://github.com/MCG-NJU/CoMo。
英文原文摘要 1624 字符
Unsupervised learning of latent motion from Internet videos is crucial for robot learning. Existing discrete methods generally mitigate the shortcut learning caused by extracting excessive static backgrounds through vector quantization with a small codebook size. However, they suffer from information loss and struggle to capture more complex and fine-grained dynamics. Moreover, there is an inherent gap between the distribution of discrete latent motion and continuous robot action, which hinders the joint learning of a unified policy. We propose CoMo, which aims to learn more precise continuous latent motion from internet-scale videos. CoMo employs an early temporal difference (Td) mechanism to increase the shortcut learning difficulty and explicitly enhance motion cues. Additionally, to ensure latent motion better captures meaningful foregrounds, we further propose a temporal contrastive learning (Tcl) scheme. Specifically, positive pairs are constructed with a small future frame temporal offset, while negative pairs are formed by directly reversing the temporal direction. The proposed Td and Tcl work synergistically and effectively ensure that the latent motion focuses better on the foreground and reinforces motion cues. Critically, CoMo exhibits strong zero-shot generalization, enabling it to generate effective pseudo action labels for unseen videos. Extensive simulated and real-world experiments show that policies co-trained with CoMo pseudo action labels achieve superior performance with both diffusion and auto-regressive architectures. The code is available at https://github.com/MCG-NJU/CoMo.