Real2Edit2Real:通过三维控制接口生成机器人演示数据
Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interface
把度量尺度三维重建 + 点云编辑得到的深度图当作视频扩散的主控制信号,配合动作/边缘/射线图合成多视角操作视频,为视觉研究者提供了一个可复用的『3D 编辑→2D 生成』数据增广管线,尤其适合空间泛化场景下的策略训练。
原文依据 we perform depth-reliable 3D editing on point clouds to generate new manipulation trajectories
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论机器人学习的最新进展由大规模数据集和强大的视觉运动策略架构推动,然而策略的鲁棒性仍受限于采集多样化演示数据的高昂成本,在操作任务的空间泛化方面尤为突出。为减少重复的数据采集,我们提出 Real2Edit2Real,一个通过三维控制接口将三维可编辑性与二维视觉数据相衔接、从而生成新演示数据的框架。该方法首先利用度量尺度的三维重建模型从多视角 RGB 观测中重建场景几何。基于重建出的几何,我们在点云上执行深度可靠的三维编辑以生成新的操作轨迹,同时对机器人位姿进行几何校正以恢复物理一致的深度,该深度作为合成新演示数据的可靠条件。最后,我们提出一个以深度为主控制信号、并以动作、边缘和射线图为辅助的多条件视频生成模型,用于合成空间增强的多视角操作视频。在四个真实世界操作任务上的实验表明,仅用 1 至 5 条源演示生成的数据所训练的策略即可匹配甚至超越用 50 条真实演示训练的策略,数据效率提升最高达 10 至 50 倍。此外,在高度与纹理编辑上的实验结果验证了该框架的灵活性与可扩展性,表明其有潜力成为统一的数据生成框架。
英文原文摘要 1554 字符
Recent progress in robot learning has been driven by large-scale datasets and powerful visuomotor policy architectures, yet policy robustness remains limited by the substantial cost of collecting diverse demonstrations, particularly for spatial generalization in manipulation tasks. To reduce repetitive data collection, we present Real2Edit2Real, a framework that generates new demonstrations by bridging 3D editability with 2D visual data through a 3D control interface. Our approach first reconstructs scene geometry from multi-view RGB observations with a metric-scale 3D reconstruction model. Based on the reconstructed geometry, we perform depth-reliable 3D editing on point clouds to generate new manipulation trajectories while geometrically correcting the robot poses to recover physically consistent depth, which serves as a reliable condition for synthesizing new demonstrations. Finally, we propose a multi-conditional video generation model guided by depth as the primary control signal, together with action, edge, and ray maps, to synthesize spatially augmented multi-view manipulation videos. Experiments on four real-world manipulation tasks demonstrate that policies trained on data generated from only 1-5 source demonstrations can match or outperform those trained on 50 real-world demonstrations, improving data efficiency by up to 10-50x. Moreover, experimental results on height and texture editing demonstrate the framework's flexibility and extensibility, indicating its potential to serve as a unified data generation framework.