从视频中学习统一潜动作空间：以动作为中心的循环一致性
Learning a Unified Latent Action Space from Videos with Action-centric Cycle Consistency
提出以动作为中心的循环一致性训练潜动作分词器，使从视频中学到的潜动作具有语义一致性并可跨本体对齐，为利用无动作标签视频数据训练机器人策略提供了可复用的表征学习手段。
原文依据 we propose the action-centric cycle consistency, aiming to establish a unified latent action space
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频数据为推进机器人学习提供了丰富来源，超越了昂贵的动作标注数据。近期方法通过为策略训练学习潜动作，展示了利用视频数据的潜力。潜动作分词器编码连续视频帧之间的潜动作，并通过当前帧和编码的潜动作重建未来帧来训练分词器。然而，连续帧的唯一配对使得未来帧重建只需很少理解转移动态，阻碍了学习语义一致的潜动作。此外，分词器通常为不同本体分配不同的潜动作子集以适应异构形态，限制了知识迁移。为克服这些限制，我们提出以动作为中心的循环一致性，旨在建立统一的潜动作空间。我们的方法从潜动作空间采样潜动作，并用视频帧解码以生成多样化的后续帧，然后通过从原始帧和生成帧预测采样动作来强制循环一致性。我们简洁的方法创建了一个具有挑战性的任务，即从当前帧和多样化的生成未来帧中学习对应的潜动作，迫使分词器发展出语义一致的动作表示。此外，采样的潜动作可应用于来自不同本体的视频帧，促进跨本体的潜动作对齐。实验表明，我们的方法在 LIBERO 基准上比 OpenVLA 提升 20.1%，在 CALVIN 基准上将平均长度从 3.27 提升至 3.93。在真实世界实验中，我们的方法保持强劲性能，提升 44%。
英文原文摘要 1839 字符
Video data provides a rich source beyond expensive action-labeled data for advancing robot learning. Recent approaches have demonstrated promising potential in leveraging video data by learning latent actions for policy training. The latent action tokenizer encodes latent actions between successive video frames, and the tokenizer is trained to reconstruct future frames using current frames and the encoded latent actions. However, the unique pairing of successive frames permits future frame reconstruction with little understanding of transition dynamics, hindering the learning of semantically consistent latent actions. Moreover, the tokenizer typically allocates distinct latent action subsets to individual embodiments to accommodate heterogeneous morphologies, constraining knowledge transfer. To overcome such limitations, we propose the action-centric cycle consistency, aiming to establish a unified latent action space. Our method samples latent actions from the latent action space and decodes them with video frames to generate diverse subsequent frames, then enforces cycle consistency by predicting the sampled actions from both original and generated frames. Our concise method creates a challenging task that learns corresponding latent actions from current frames and diverse generated future frames, compelling the tokenizer to develop semantically consistent action representations. Additionally, sampled latent actions can be applied to video frames from distinct embodiments, facilitating the alignment of latent actions across embodiments. Experiments demonstrate that our approach achieves a 20.1% improvement over OpenVLA on the LIBERO benchmark and increases the average length from 3.27 to 3.93 on the CALVIN benchmark. In real-world experiments, our method maintains strong performance with a 44% improvement.