灵巧世界模型
Dexterous World Models
将静态3D场景渲染与第一人称手部网格渲染作为条件输入视频扩散模型，实现动作驱动的场景交互视频生成，为具身交互与数字孪生提供可复用的条件生成框架。
原文依据 an scene-action-conditioned video diffusion model enabling embodied interaction within static 3D scenes
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期三维重建的进展使得从日常环境中轻松创建逼真的数字孪生成为可能。然而，当前的数字孪生在很大程度上仍是静态的——仅限于导航和视图合成，缺乏具身交互性。为弥补这一差距，我们提出灵巧世界模型（DWM），一种以场景和动作为条件的视频扩散模型，能够在静态三维场景中实现具身交互。给定静态三维场景渲染和第一人称手部运动序列，DWM生成描绘合理人-场景交互的时间连贯视频。我们的方法将视频生成条件设定为：（1）遵循指定相机轨迹的静态场景渲染，以确保空间一致性；（2）第一人称手部网格渲染，在第一人称视角中编码几何与运动线索，以直接建模动作条件动态。我们在合成人-场景交互数据集和真实世界物体操作数据集上训练模型，然后在合成与真实的第一人称基准上评估。实验表明，DWM能够实现真实且物理合理的交互，例如抓取、打开或移动物体，同时保持相机和场景一致性。该框架为基于视频扩散的交互式数字孪生迈出了第一步，使得从第一人称动作出发的具身模拟和三维场景交互成为可能。
英文原文摘要 1411 字符
Recent progress in 3D reconstruction has made it easy to create realistic digital twins from everyday environments. However, current digital twins remain largely static--limited to navigation and view synthesis without embodied interactivity. To bridge this gap, we introduce Dexterous World Model (DWM), an scene-action-conditioned video diffusion model enabling embodied interaction within static 3D scenes. Given a static 3D scene rendering and an egocentric hand motion sequence, DWM generates temporally coherent videos depicting plausible human-scene interactions. Our approach conditions video generation on (1) static scene renderings following a specified camera trajectory to ensure spatial consistency, and (2) egocentric hand mesh renderings that encode both geometry and motion cues in the egocentric view to model action-conditioned dynamics directly. We train our model on a synthetic human-scene interaction dataset and real-world object manipulation dataset, then evaluate it across both synthetic and real-world egocentric benchmarks. Experiments demonstrate that DWM enables realistic, physically grounded interactions, such as grasping, opening, or moving objects, while maintaining camera and scene consistency. This framework establishes the first step toward video diffusion-based interactive digital twins, enabling embodied simulation and 3D scene interactivity from egocentric actions.