ForeHOI：从日常手物交互视频前馈重建三维物体
ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos
把 2D 掩码修复与 3D 形状补全放进同一个前馈框架联合预测，用 2D/3D 之间的信息交换解决手物严重遮挡下的物体重建，可迁移到任何被遮挡物体的单目重建与补全任务。
原文依据 the joint prediction of 2D mask inpainting and 3D shape completion in a feed-forward framework can effectively address the problem of severe occlusion
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大量记录日常手物交互的单目视频为具身智能提供了宝贵资源。尽管从野外视频中进行三维手部重建已取得显著进展，但由于严重的遮挡以及相机、手与物体之间复杂且耦合的运动，重建其中涉及的物体仍然具有挑战性。本文提出 ForeHOI，一种新颖的前馈模型，可在约一分钟推理时间内直接从单目手物交互视频重建三维物体几何，且无需任何预处理步骤。我们的关键洞见是：在前馈框架中联合预测二维掩码图像修复与三维形状补全，能够有效应对单目手持物体视频中的严重遮挡问题，从而取得优于基于优化的方法的结果。二维与三维形状补全之间的信息交换提升了整体重建质量，使该框架能够有效处理严重的手物遮挡。此外，为支持模型训练，我们贡献了首个大规模、高保真的手物交互合成数据集，并配有全面的标注。大量实验表明，ForeHOI 在物体重建上达到最优性能，相比先前方法显著领先，并带来约 100 倍的加速。代码与数据见：https://github.com/Tao-11-chen/ForeHOI。
英文原文摘要 1526 字符
The ubiquity of monocular videos capturing daily hand-object interactions presents a valuable resource for embodied intelligence. While 3D hand reconstruction from in-the-wild videos has seen significant progress, reconstructing the involved objects remains challenging due to severe occlusions and the complex, coupled motion of the camera, hands, and object. In this paper, we introduce ForeHOI, a novel feed-forward model that directly reconstructs 3D object geometry from monocular hand-object interaction videos within one minute of inference time, eliminating the need for any pre-processing steps. Our key insight is that, the joint prediction of 2D mask inpainting and 3D shape completion in a feed-forward framework can effectively address the problem of severe occlusion in monocular hand-held object videos, thereby achieving results that outperform the performance of optimization-based methods. The information exchanges between the 2D and 3D shape completion boosts the overall reconstruction quality, enabling the framework to effectively handle severe hand-object occlusion. Furthermore, to support the training of our model, we contribute the first large-scale, high-fidelity synthetic dataset of hand-object interactions with comprehensive annotations. Extensive experiments demonstrate that ForeHOI achieves state-of-the-art performance in object reconstruction, significantly outperforming previous methods with around a 100x speedup. Code and data are available at: https://github.com/Tao-11-chen/ForeHOI.