HandWorld：以手为中心的统一视频动作生成
HandWorld: Hand-Centric Unified Video Action Generation
用双分支条件网络把视频域与动作域对齐成共享条件，再由两个解耦的扩散 Transformer 分别生成，这种「共享条件 + 分域解耦生成」的范式可直接迁移到其他需要联合生成视觉与结构化信号的设定（如轨迹+视频、音频+视频）。MANO 渲染手部作为中间输入提升跨域一致性的做法，对手物交互视频生成与动作预测都有参考价值。
原文依据 two decoupled diffusion transformers are trained to predict in their respective domain
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论手物交互构成了人类与世界交互的基础。理解手部动作与第一人称视频之间的联系，对于使具身智能体能够像人类一样感知、模拟和规划至关重要。然而，由于手部动作与第一人称视频之间存在非线性关系，跨二者进行学习与预测颇具挑战。在本工作中，我们提出 HandWorld，一个聚焦于手物交互、并联合建模第一人称视频与手部动作的统一生成框架。HandWorld 通过双分支条件网络学习共享的跨域条件，该网络整合来自视频域与动作域的信息。MANO 渲染的手部表示被作为中间输入引入，以进一步增强跨域一致性。在共享表示作为条件的基础上，两个解耦的扩散 Transformer 分别在各自域中进行预测。一种灵活的训练策略使模型能够跨多种任务配置进行学习，包括动作预测与可控视频生成。在大规模第一人称手物交互数据集上的实验表明，HandWorld 实现了高保真的视频合成与准确的动作预测，在多种场景下优于现有基线。
英文原文摘要 1296 字符
Hand-object interaction forms the foundation of how humans interact with the world. Understanding the connection between hand action and egocentric video is essential for enabling embodied agents to perceive, simulate, and plan like humans. However, it is challenging to learn and predict across hand actions and egocentric videos due to their non-linear relationship. In this work, we introduce HandWorld, a unified generative framework that focuses on hand-object interaction and jointly models egocentric videos and hand actions. HandWorld learns shared cross-domain conditions through a dual-branch condition network that integrates information from both video and action domains. MANO-rendered hand representation is incorporated as an intermediate input to further enhance cross-domain coherence. Conditioned on the shared representation, two decoupled diffusion transformers are trained to predict in their respective domain. A flexible training strategy enables the model to learn across diverse task configurations, including action forecasting and controllable video generation. Experiments on large-scale egocentric HOI datasets demonstrate that HandWorld achieves high-fidelity video synthesis and accurate action prediction, outperforming existing baselines across diverse scenarios.