HVG-3D：打通真实与仿真域的三维条件手物交互视频合成
HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesis
将 3D ControlNet 接入扩散视频生成主干，用显式三维几何与运动线索替代 2D 控制信号，为视觉研究者提供了一条把仿真 3D 数据与真实单图结合、实现时空可控视频合成的可复用路径。
原文依据 a diffusion-based architecture augmented with a 3D ControlNet, which encodes geometric and motion cues from 3D inputs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期方法在手物交互视频合成的视觉质量上取得了显著进展。然而，大多数方法依赖缺乏空间表达能力的二维控制信号，并限制了合成三维条件数据的利用。为解决这些局限，我们提出 HVG-3D，一个以显式三维表示为条件的、统一的三维感知手物交互（HOI）视频合成框架。为此，我们实现了一种由三维 ControlNet 增强的扩散架构，该架构从三维输入中编码几何与运动线索，从而在视频合成过程中实现显式的三维推理，并给出了相应的训练与推理设置。为实现高质量合成，HVG-3D 设计有两个核心组件：（i）三维感知的 HOI 视频生成扩散架构，从三维输入中编码几何与运动线索以进行显式三维推理；（ii）用于构建输入与条件信号的混合流程，在训练与推理过程中实现灵活而精确的控制。在推理阶段，给定单张真实图像以及来自仿真或真实数据的三维控制信号，HVG-3D 可生成高保真、时序一致的视频，并具备精确的空间与时序控制能力。在 TASTE-Rob 数据集上的实验表明，HVG-3D 在空间保真度、时序连贯性与可控性上达到最优，同时能够有效利用真实数据与仿真数据。
英文原文摘要 1462 字符
Recent methods have made notable progress in the visual quality of hand-object interaction video synthesis. However, most approaches rely on 2D control signals that lack spatial expressiveness and limit the utilization of synthetic 3D conditional data. To address these limitations, we propose HVG-3D, a unified framework for 3D-aware hand-object interaction (HOI) video synthesis conditioned on explicit 3D representations. To achieve a diffusion-based architecture augmented with a 3D ControlNet, which encodes geometric and motion cues from 3D inputs to enable explicit 3D reasoning during video synthesis, as well as the corresponding training and inference setting. To achieve high-quality synthesis, HVG-3D is designed with two core components: (i) a 3D-aware HOI video generation diffusion architecture that encodes geometric and motion cues from 3D inputs for explicit 3D reasoning; and (ii) a hybrid pipeline for constructing input and condition signals, enabling flexible and precise control during both training and inference. During inference, given a single real image and a 3D control signal from either simulation or real data, HVG-3D generates high-fidelity, temporally consistent videos with precise spatial and temporal control. Experiments on the TASTE-Rob dataset demonstrate that HVG-3D achieves state-of-the-art spatial fidelity, temporal coherence, and controllability, while enabling effective utilization of both real and simulated data.