通过生成并模仿 2D 运动学习控制物理仿真的 3D 角色
Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motions
把视频中提取的 2D 运动当作物理仿真角色控制策略的监督信号，绕开 3D 运动重建的物理不合理性；其「多视角 2D 跟踪隐式学出 3D 结构」的思路，对做单目视频驱动角色动画、以及用 2D 标注替代 3D 标注的视觉研究者是可复用的技术路线。
原文依据 we train a general single-view tracking policy capable of imitating arbitrary reference motions in a physics simulation using exclusively 2D data
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论相比动作捕捉数据，视频数据在学习 3D 角色控制器时更具成本效益，但利用视频生成真实且物理合理的运动仍然具有挑战性。已有方法通常依赖现成的运动重建技术来提取 3D 运动学轨迹，以用于基于物理的模仿。然而，这类方法从根本上受限于所估计运动的物理不合理性，并且缺乏一个统一框架来处理非人类主体以及复杂的人—物交互（HOI）。我们通过引入 Mimic2DM 来应对这一挑战，这是一个新颖的运动模仿框架，直接且仅从视频中提取的 2D 运动学习控制策略。通过构建 2D 运动跟踪目标，我们训练出一个通用的单视角跟踪策略，它能够仅使用 2D 数据在物理仿真中模仿任意参考运动。此外，通过在来自不同视角的 2D 运动上训练，我们的视角无关策略隐式地学习到 3D 运动结构，使其仅通过聚合多个视角即可执行完整的三维运动跟踪。为扩展这一能力，我们将该跟踪策略与一个基于自回归 Transformer 的 2D 运动生成器相结合，构建出一个具备长时程运动合成与交互控制能力的分层控制框架。我们表明，所提方法具有通用性，能够在舞蹈、足球带球和动物运动等多个领域中有效合成物理合理且多样的运动，而完全不依赖显式的 3D 运动数据。
英文原文摘要 1621 字符
Video data is more cost-effective than motion capture data for learning 3D character controllers, yet using it to generate realistic and physically plausible motions remains challenging. Previous approaches typically rely on off-the-shelf motion reconstruction techniques to extract 3D kinematic trajectories for physics-based imitation. However, these approaches are fundamentally limited by the physical implausibility of the estimated motions and lack a unified framework for handling non-human subjects and complex human-object interactions (HOI). We tackle this challenge by introducing Mimic2DM, a novel motion imitation framework that learns the control policy directly and solely from 2D motions extracted from videos. By formulating a 2D motion tracking objective, we train a general single-view tracking policy capable of imitating arbitrary reference motions in a physics simulation using exclusively 2D data. Furthermore, by training on 2D motions from diverse viewpoints, our view-agnostic policy implicitly learns 3D motion structures, allowing it to perform full 3D motion tracking simply by aggregating multiple views. To extend this capability, we integrate the tracking policy with an autoregressive transformer-based 2D motion generator, constructing a hierarchical control framework capable of long-horizon motion synthesis and interactive control. We show that the proposed approach is versatile and effectively synthesizes physically plausible and diverse motions across a range of domains, including dancing, soccer dribbling, and animal locomotion, without any reliance on explicit 3D motion data.