AnyLift：借助 2D 扩散从互联网视频扩展运动重建
AnyLift: Scaling Motion Reconstruction from Internet Videos via 2D Diffusion
用 2D 扩散模型在多视图合成数据上做相机条件建模，把互联网单目视频中的 2D 关键点提升为世界坐标系下的全局一致 3D 人体运动与 HOI，为动态相机下的运动重建提供了可复用的两阶段范式。
原文依据 a camera-conditioned multi-view 2D motion diffusion model is trained on the domain-specific synthetic data to recover 3D human motion
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从互联网视频中重建三维人体运动与人-物交互（HOI），是构建大规模人类行为数据集的基础性一步。现有方法在动态相机下难以恢复全局一致的三维运动，对于当前动作捕捉数据集中覆盖不足的运动类型尤其如此，并且在恢复三维空间中连贯的人-物交互时还面临额外困难。我们提出一个利用 2D 扩散的两阶段框架，从互联网视频中重建三维人体运动与 HOI。在第一阶段，我们为每个领域合成多视图 2D 运动数据，利用从互联网视频中提取的 2D 关键点来纳入现有动作捕捉数据集中很少出现的人体运动。在第二阶段，我们在领域特定的合成数据上训练一个以相机为条件的多视图 2D 运动扩散模型，以在世界空间中恢复三维人体运动与三维 HOI。我们在包含体操等挑战性运动的互联网视频以及野外 HOI 视频上验证了该方法的有效性，并表明其在生成逼真的人体运动与人-物交互方面优于先前工作。
英文原文摘要 1194 字符
Reconstructing 3D human motion and human-object interactions (HOI) from Internet videos is a fundamental step toward building large-scale datasets of human behavior. Existing methods struggle to recover globally consistent 3D motion under dynamic cameras, especially for motion types underrepresented in current motion-capture datasets, and face additional difficulty recovering coherent human-object interactions in 3D. We introduce a two-stage framework leveraging 2D diffusion that reconstructs 3D human motion and HOI from Internet videos. In the first stage, we synthesize multi-view 2D motion data for each domain, leveraging 2D keypoints extracted from Internet videos to incorporate human motions that rarely appear in existing MoCap datasets. In the second stage, a camera-conditioned multi-view 2D motion diffusion model is trained on the domain-specific synthetic data to recover 3D human motion and 3D HOI in the world space. We demonstrate the effectiveness of our method on Internet videos featuring challenging motions such as gymnastics, as well as in-the-wild HOI videos, and show that it outperforms prior work in producing realistic human motion and human-object interaction.