无需见过组合式 4D 场景即可推断
Inferring Compositional 4D Scenes without Ever Seeing One
提出时空注意力解耦训练与注意力混合推理机制，使模型无需 4D 组合训练数据即可从单目视频重建多物体交互的 4D 场景，为动态场景重建提供可复用的训练策略。
原文依据 We achieve this by a carefully designed training of spatial and temporal attentions on 2D video input.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现实世界中的场景通常由多个静态和动态物体组成。在野外捕获它们的四维结构、组成和时空配置虽然极其有趣，但同样困难。因此，现有工作通常一次只关注一个物体，同时依赖某种类别特定的参数化形状模型来处理动态物体。这可能导致场景配置不一致，并且局限于已建模的物体类别。我们提出 COM4D（组合式 4D），一种仅使用静态多物体或动态单物体监督，就能一致且联合地预测 4D/3D 物体的结构和时空配置的方法。我们通过在 2D 视频输入上精心设计的空间和时间注意力训练来实现这一点。训练被解耦为一方面从物体组成中学习，另一方面从视频中的单物体动态中学习，从而完全避免依赖 4D 组合训练数据。在推理时，我们提出的注意力混合机制结合这些独立学习到的注意力，无需任何 4D 组合示例。通过在空间和时间推理之间交替，COM4D 直接从单目视频中重建具有多个交互物体的完整且持久的 4D 场景。尽管 COM4D 完全是数据驱动的，但它在现有的 4D 物体和组合式 3D 重建这两个独立问题上均提供了最优结果。
英文原文摘要 1503 字符
Scenes in the real world are often composed of several static and dynamic objects. Capturing their 4-dimensional structures, composition and spatio-temporal configuration in-the-wild, though extremely interesting, is equally hard.Therefore, existing works often focus on one object at a time, while relying on some category-specific parametric shape model for dynamic objects. This can lead to inconsistent scene configurations, in addition to being limited to the modeled object categories. We propose COM4D (Compositional 4D), a method that consistently and jointly predicts the structure and spatio-temporal configuration of 4D/3D objects using only static multi-object or dynamic single object supervision. We achieve this by a carefully designed training of spatial and temporal attentions on 2D video input. The training is disentangled into learning from object compositions on the one hand, and single object dynamics throughout the video on the other, thus completely avoiding reliance on 4D compositional training data.At inference time, our proposed attention mixing mechanism combines these independently learned attentions, without requiring any 4D composition examples.By alternating between spatial and temporal reasoning, COM4D reconstructs complete and persistent 4D scenes with multiple interacting objects directly from monocular videos.COM4D provides state-of-the-art results in existing separate problems of 4D object and composed 3D reconstruction despite being purely data-driven.