强化学习与具身智能 · CVPR2026 ·2026.06.03

RoboWheel:面向跨形态机器人学习的真实世界人类演示数据引擎

RoboWheel: A Data Engine from Real-World Human Demonstrations for Cross-Embodiment Robotic Learning

Yuhong Zhang、Zihan Gao、Shengpeng Li、Ling-Hao Chen、Kaisheng Liu、Runqing Cheng 等 15 人

与你有关的部分

从单目RGB(D)视频重建手物交互并重定向为跨形态机器人动作,为视觉研究者提供了一条将人类演示视频转化为机器人训练数据的可复用管线,其中基于强化学习的接触与穿透约束优化器可用于提升手物姿态估计的物理合理性。

原文依据 we perform high-precision HOI reconstruction and enforce physical plausibility via a reinforcement learning (RL) optimizer
可迁移到 手物交互重建视觉-语言-动作学习模仿学习数据增强

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

我们提出 RoboWheel,一个将人类手物交互(HOI)视频转化为面向跨形态机器人学习的训练就绪监督的数据引擎。从单目 RGB/RGB-D 输入出发,我们执行高精度 HOI 重建,并通过一个强化学习(RL)优化器在接触与穿透约束下精化手物相对位姿,从而保证物理合理性。随后,重建得到的富含接触的轨迹被重定向到跨形态实体——具有简单末端执行器的机械臂、灵巧手以及人形机器人——产生可执行的动作与 rollout。为扩展覆盖范围,我们构建了一个基于 Isaac Sim 的仿真增强框架,采用多样化的域随机化(形态、轨迹、物体检索、背景纹理、手部运动镜像),在保持空间关系与物理合理性的同时丰富轨迹与观测的分布。整个数据管线形成从视频→重建→重定向→增强→数据获取的端到端流程。我们在主流的视觉-语言-动作(VLA)与模仿学习架构上验证了该数据,结果表明我们的管线生成的轨迹与遥操作轨迹一样稳定,并能带来可比的持续性能增益。据我们所知,这首次提供了 HOI 模态可作为机器人学习有效监督的定量证据。与遥操作相比,RoboWheel 是轻量级的:仅需单个单目 RGB(D) 相机即可提取通用的、与形态无关的运动表示,并可灵活地跨形态重定向。我们进一步构建了一个大规模多模态数据集,结合多相机采集、单目视频与公开 HOI 语料库,用于训练和评估具身模型。

英文原文摘要 1850 字符

We introduce Robowheel, a data engine that converts human hand-object interaction (HOI) videos into training-ready supervision for cross-morphology robotic learning. From monocular RGB/RGB-D inputs, we perform high-precision HOI reconstruction and enforce physical plausibility via a reinforcement learning (RL) optimizer that refines hand-object relative poses under contact and penetration constraints. The reconstructed, contact-rich trajectories are then retargeted to cross-embodiments, robot arms with simple end-effectors, dexterous hands, and humanoids, yielding executable actions and rollouts. To scale coverage, we build a simulation-augmented framework on Isaac Sim with diverse domain randomization (embodiments, trajectories, object retrieval, background textures, hand motion mirroring), which enriches the distributions of trajectories and observations while preserving spatial relationships and physical plausibility. The entire data pipeline forms an end-to-end pipeline from video - reconstruction - retargeting - augmentation - data acquisition.We validate the data on mainstream vision-language-action (VLA) and imitation learning architectures, demonstrating that trajectories produced by our pipeline are as stable as those from teleoperation and yield comparable continual performance gains. To our knowledge, this provides the first quantitative evidence that HOI modalities can serve as effective supervision for robotic learning. Compared with teleoperation, Robowheel is lightweight: a single monocular RGB(D) camera is sufficient to extract a universal, embodiment-agnostic motion representation that could be flexibly retargeted across embodiments. We further assemble a large-scale multimodal dataset combining multi-camera captures, monocular videos, and public HOI corpora for training and evaluating embodied models.

标签

会议 CVPR2026
页码 6664-6674
Bibkey
来源 CVF Open Access