IGen：从开放世界图像可扩展地生成机器人学习数据
IGen: Scalable Data Generation for Robot Learning from Open-World Images
把开放世界单张图像转成可用于操作策略训练的视频-动作对：先用 3D 场景表示重建，再由 VLM 规划并输出 SE(3) 末端位姿，最后渲染时序一致的观测。做视觉-动作数据合成、VLA 预训练或 sim-to-real 的工作可以直接复用这条「图像→3D→位姿→视频」的流水线来扩充训练集。
原文依据 generate low-level actions as SE(3) end-effector pose sequences
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论通用机器人策略的兴起对大规模训练数据产生了指数级需求。然而，在机器人本体上采集数据劳动密集，且往往局限于特定环境。相比之下，开放世界图像捕获了真实世界场景的巨大多样性，这些场景天然与机器人操作任务相契合，为低成本、大规模的机器人数据获取提供了有前景的途径。尽管具备这一潜力，缺少与之关联的机器人动作阻碍了开放世界图像在机器人学习中的实际使用，使这一丰富的视觉资源在很大程度上未被利用。为弥合这一差距，我们提出 IGen，一个能够从开放世界图像可扩展地生成逼真视觉观测与可执行动作的框架。IGen 首先将非结构化的 2D 像素转换为适合场景理解与操作的结构化 3D 场景表示。随后，它利用视觉语言模型的推理能力，将场景特定的任务指令转化为高层规划，并生成以 SE(3) 末端执行器位姿序列表示的低层动作。基于这些位姿，它合成动态的场景演化，并渲染时序一致的视觉观测。实验验证了 IGen 生成的视觉运动数据具有高质量，并表明仅在 IGen 合成数据上训练的策略即可达到与在真实世界数据上训练相当的性能。这凸显了 IGen 在从开放世界图像为通用机器人策略训练提供可扩展数据生成方面的潜力。IGen 的代码将公开。
英文原文摘要 1587 字符
The rise of generalist robotic policies has created an exponential demand for large-scale training data. However, on-robot data collection is labor-intensive and often limited to specific environments. In contrast, open-world images capture a vast diversity of real-world scenes that naturally align with robotic manipulation tasks, offering a promising avenue for low-cost, large-scale robot data acquisition. Despite this potential, the lack of associated robot actions hinders the practical use of open-world images for robot learning, leaving this rich visual resource largely unexploited. To bridge this gap, we propose IGen, a framework that scalably generates realistic visual observations and executable actions from open-world images. IGen first converts unstructured 2D pixels into structured 3D scene representations suitable for scene understanding and manipulation. It then leverages the reasoning capabilities of vision-language models to transform scene-specific task instructions into high-level plans and generate low-level actions as SE(3) end-effector pose sequences. From these poses, it synthesizes dynamic scene evolution and renders temporally coherent visual observations. Experiments validate the high quality of visuomotor data generated by IGen, and show that policies trained solely on IGen-synthesized data achieve performance comparable to those trained on real-world data. This highlights the potential of IGen to support scalable data generation from open-world images for generalist robotic policy training. Code for IGen will be made publicly available.