Elastic3D:基于引导式潜变量解码的可控立体视频转换
Elastic3D: Controllable Stereo Video Conversion with Guided Latent Decoding
用条件潜扩散直接生成右视图、绕开显式深度估计与 warping,并设计引导式 VAE 解码器保证锐利与极线一致;其标量视差调节旋钮为立体/新视角生成提供了推理期可控性的可复用范式。
原文依据 a novel, guided VAE decoder that ensures sharp and epipolar-consistent stereo video output
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论对沉浸式 3D 内容日益增长的需求,催生了自动化的单目到立体视频转换。本文提出一种可控、直接的端到端方法,将普通视频升级为双目视频。该方法基于(条件)潜扩散,避免了显式深度估计与 warping 带来的伪影。其高质量立体视频输出的关键在于一种新颖的引导式变分自编码器解码器,可确保输出立体视频锐利且极线一致。此外,该方法通过一个直观的标量调节旋钮,使用户能够在推理时控制立体效果的强度(即视差范围)。在三个真实世界立体视频数据集上的实验表明,该方法优于传统的基于 warping 的基线以及近期无 warping 的基线,为可靠、可控的立体视频转换树立了新标准。视频样例请见项目主页:elastic3d.github.io
英文原文摘要 960 字符
The growing demand for immersive 3D content calls for automated monocular-to-stereo video conversion. We present a controllable, direct end-to-end method for upgrading a conventional video to a binocular one. Our approach, based on (conditional) latent diffusion, avoids artifacts due to explicit depth estimation and warping. The key to its high-quality stereo video output is a novel, guided VAE decoder that ensures sharp and epipolar-consistent stereo video output. Moreover, our method gives the user control over the strength of the stereo effect (respectively, the disparity range) at inference time, via an intuitive, scalar tuning knob. Experiments on three different datasets of real-world stereo videos show that our method outperforms both traditional warping-based and recent warping-free baselines and sets a new standard for reliable, controllable stereo video conversion. Please check the project page for the video samples: elastic3d.github.io