CineScene：以隐式三维作为电影级视频生成的有效场景表示
CineScene: Implicit 3D as Effective Scene Representation for Cinematic Video Generation
用 VGGT 把静态场景图像编码成隐式三维感知特征，再以额外上下文拼接的方式注入预训练文生视频模型，实现相机轨迹可控且场景一致的视频合成；这套「隐式三维先验 + 上下文拼接」的条件注入思路可迁移到任何需要几何一致性的可控视频生成任务。
原文依据 injects spatial priors into a pretrained text-to-video generation model by additional context concatenation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论电影级视频制作需要控制场景与主体的构图以及相机运动，但实拍成本高昂，因为需要搭建实体布景。为解决这一问题，我们提出了场景上下文解耦的电影级视频生成任务：给定静态环境的多张图像，目标是合成包含动态主体、同时保持底层场景一致性并遵循用户指定相机轨迹的高质量视频。我们提出 CineScene，一个利用隐式三维感知场景表示进行电影级视频生成的框架。我们的关键创新是一种新颖的上下文条件机制，以隐式方式注入三维感知特征：通过 VGGT 将场景图像编码为视觉表示，CineScene 以额外上下文拼接的方式向预训练文本到视频生成模型注入空间先验，从而在保持场景一致与动态主体的同时实现相机可控的视频合成。为进一步增强模型的鲁棒性，我们在训练时对输入场景图像引入了一种简单而有效的随机打乱策略。为缓解训练数据不足，我们使用 Unreal Engine 5 构建了一个场景解耦数据集，其中包含有动态主体与无动态主体的成对视频、表示底层静态场景的全景图像及其相机轨迹。实验表明，CineScene 在场景一致的电影级视频生成上取得了最优性能，能够处理大幅度的相机运动，并在多样环境中展现出泛化能力。
英文原文摘要 1634 字符
Cinematic video production requires control over scene-subject composition and camera movement, but live-action shooting remains costly due to the need for constructing physical sets. To address this, we introduce the task of cinematic video generation with decoupled scene context: given multiple images of a static environment, the goal is to synthesize high-quality videos featuring dynamic subject while preserving the underlying scene consistency and following a user-specified camera trajectory. We present CineScene, a framework that leverages implicit 3D-aware scene representation for cinematic video generation. Our key innovation is a novel context conditioning mechanism that injects 3D-aware features in an implicit way: By encoding scene images into visual representations through VGGT, CineScene injects spatial priors into a pretrained text-to-video generation model by additional context concatenation, enabling camera-controlled video synthesis with consistent scenes and dynamic subjects. To further enhance the model's robustness, we introduce a simple yet effective random-shuffling strategy for the input scene images during training. To address the lack of training data, we construct a scene-decoupled dataset with Unreal Engine 5, containing paired videos of scenes with and without dynamic subjects, panoramic images representing the underlying static scene, along with their camera trajectories. Experiments show that CineScene achieves state-of-the-art performance in scene-consistent cinematic video generation, handling large camera movements and demonstrating generalization across diverse environments.