PAM：面向仿真到现实手物交互视频生成的姿态-外观-运动引擎
PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generation
把手物交互的位姿、外观、运动统一进一个可控视频生成引擎，并用生成视频作为合成数据反哺下游手部姿态估计，为视觉研究者提供了「生成即数据增强」的闭环范式，以及深度/分割/关键点多条件输入的具体消融结论。
原文依据 augmenting training with 3,400 synthetic videos (207k frames) allows a model trained on only 50% of the real data plus our synthetic data to match the 100% real baseline
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论手物交互（HOI）重建与合成正成为具身智能与 AR/VR 的核心议题。然而，尽管进展迅速，现有 HOI 生成研究仍分散在三条彼此割裂的路线中：（1）仅姿态合成，只预测 MANO 轨迹而不产生像素；（2）单图像 HOI 生成，从掩码或二维线索中臆想外观但缺乏动态；（3）视频生成方法，需要整段姿态序列与真实首帧作为输入，因而无法真正实现仿真到现实部署。受前人工作思想的启发，我们认为 HOI 生成需要一个统一引擎，将姿态、外观与运动整合进同一个连贯框架。为此我们提出 PAM：一个用于可控 HOI 视频生成的姿态-外观-运动引擎。该引擎的性能由以下结果验证：（1）在 DexYCB 上，我们取得 29.13 的 FVD（InterDyn 为 38.83）与 19.37 mm 的 MPJPE（CosHand 为 30.05 mm），同时生成 480x720 的更高分辨率视频，而基线为 256x256/256x384。（2）在 OAKINK2 上，我们的完整多条件模型将 FVD 从 68.76 改善至 46.31。（3）在 DexYCB 上对输入条件的消融表明，组合深度、分割与关键点能稳定取得最佳结果。（4）在使用 SimpleHand 的下游手部姿态估计任务中，用 3,400 段合成视频（207k 帧）增强训练，可使仅用 50% 真实数据加我们合成数据训练的模型追平 100% 真实数据的基线。
英文原文摘要 1547 字符
Hand-object interaction (HOI) reconstruction and synthesis are becoming central to embodied AI and AR/VR. Yet, despite rapid progress, existing HOI generation research remains fragmented across three disjoint tracks: (1) pose-only synthesis that predicts MANO trajectories without producing pixels; (2) single-image HOI generation that hallucinates appearance from masks or 2D cues but lacks dynamics; and (3) video generation methods that require both the entire pose sequence and the ground-truth first frame as inputs, preventing true sim-to-real deployment. Inspired by the philosophy of previous work, we think that HOI generation requires a unified engine that brings together pose, appearance, and motion within one coherent framework. Thus we introduce PAM: a Pose-Appearance-Motion Engine for controllable HOI video generation. The performance of our engine is validated by: (1) On DexYCB, we obtain an FVD of 29.13 (vs. 38.83 for InterDyn), and MPJPE of 19.37 mm (vs. 30.05 mm for CosHand), while generating higher-resolution 480x720 videos compared to 256x256/256x384 baselines. (2) On OAKINK2, our full multi-condition model improves FVD from 68.76 - 46.31. (3) An ablation over input conditions on DexYCB shows that combining depth, segmentation, and keypoints consistently yields the best results. (4) For a downstream hand pose estimation task using SimpleHand, augmenting training with 3,400 synthetic videos (207k frames) allows a model trained on only 50% of the real data plus our synthetic data to match the 100% real baseline.