从第一视角步行游览视频生成无人环境漫游
Generating Humanless Environment Walkthroughs from Egocentric Walking Tour Videos
用半合成视频对（纯背景片段 + 带模拟阴影的行走行人合成片段）微调 Casper 视频扩散模型做人与阴影联合修复，为从众包第一视角视频中清洗出可用于 3D/4D 重建的干净素材提供了可复用的数据构造与微调范式。
原文依据 fine-tune the state-of-the-art Casper video diffusion model for object and effects inpainting
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论第一视角步行游览视频为构建全球各地环境的丰富多样视觉模型提供了丰富的图像数据来源。然而，由于人群密集以及眼平高度的相机视角，这些视频画面中大量出现的人体会削弱其在环境建模应用中的价值。我们致力于应对这一挑战，开发一种生成算法，能够从步行游览视频中真实地移除（即修复）人体及其相关的阴影效应。我们方法的关键在于构建一个丰富的半合成视频片段对数据集来训练该生成模型。数据集中的每一对由一个仅含环境的背景片段和一个将行走的人体连同模拟阴影叠加在背景上的合成片段组成。我们从全球各地真实的第二视角步行游览视频中随机采集前景与背景成分，以保持视觉多样性。随后，我们使用该数据集对用于物体与效应修复的最优 Casper 视频扩散模型进行微调，并证明所得模型在从人体密集出现、背景复杂的步行游览片段中移除人体方面，在定性和定量上均远优于 Casper。最后，我们展示所生成的片段可用于构建成功的城市地点 3D/4D 模型。
英文原文摘要 1406 字符
Egocentric walking tour videos provide a rich source of image data to develop rich and diverse visual models of environments around the world. However, the significant presence of humans in frames of these videos due to crowds and eye-level camera perspectives mitigates their usefulness in environment modeling applications. We focus on addressing this challenge by developing a generative algorithm that can realistically remove (i.e., inpaint) humans and their associated shadow effects from walking tour videos. Key to our approach is the construction of a rich semi-synthetic dataset of video clip pairs to train this generative model. Each pair in the dataset consists of an environment-only background clip, and a composite clip of walking humans with simulated shadows overlaid on the background. We randomly sourced both foreground and background components from real egocentric walking tour videos around the world to maintain visual diversity. We then used this dataset to fine-tune the state-of-the-art Casper video diffusion model for object and effects inpainting, and demonstrate that the resulting model performs far better than Casper both qualitatively and quantitatively at removing humans from walking tour clips with significant human presence and complex backgrounds. Finally, we show that the resulting generated clips can be used to build successful 3D/4D models of urban locations.