面向可控视频生成的统一相机位置编码
Unified Camera Positional Encoding for Controlled Video Generation
提出 Relative Ray Encoding 把 6-DoF 位姿、内参和镜头畸变统一编码为几何一致的射线表示，并以不到 1% 可训练参数的轻量空间注意力适配器注入预训练视频 DiT，可直接复用于多视角生成、新视角合成和 3D 感知骨干的相机条件注入。
原文依据 a geometry-consistent representation that unifies complete camera information, including 6-DoF poses, intrinsics, and lens distortions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论Transformer 已成为 3D 感知、视频生成以及面向自动驾驶和具身智能的世界模型中的通用骨干网络，而理解相机几何对于将视觉观测锚定到三维空间至关重要。然而，现有的相机编码方法往往依赖简化的针孔假设，限制了其在真实相机多样的内参与镜头畸变下的泛化能力。我们提出相对射线编码（Relative Ray Encoding），这是一种几何一致的表示，统一了完整的相机信息，包括 6 自由度位姿、内参和镜头畸变。为评估其在多样可控性需求下的能力，我们采用相机控制的文本到视频生成作为测试任务。在该设定下，我们进一步发现俯仰角和横滚角是适用于绝对朝向编码（Absolute Orientation Encoding）的两个分量，从而实现对初始相机朝向的完整控制。这些设计共同构成 UCPE（统一相机位置编码），它通过一个轻量的空间注意力适配器集成到预训练视频扩散 Transformer 中，仅增加不到 1% 的可训练参数，同时实现最优的相机可控性与视觉保真度。为便于系统化的训练与评估，我们构建了一个覆盖广泛相机运动与镜头类型的大型视频数据集。大量实验验证了 UCPE 在相机可控视频生成中的有效性，并凸显了其作为 Transformer 通用相机表示在未来多视角、视频和 3D 任务中的潜力。
英文原文摘要 1616 字符
Transformers have emerged as a universal backbone across 3D perception, video generation, and world models for autonomous driving and embodied AI, where understanding camera geometry is essential for grounding visual observations in three-dimensional space. However, existing camera encoding methods often rely on simplified pinhole assumptions, restricting generalization across the diverse intrinsics and lens distortions in real-world cameras. We introduce **Relative Ray Encoding**, a geometry-consistent representation that unifies complete camera information, including 6-DoF poses, intrinsics, and lens distortions. To evaluate its capability under diverse controllability demands, we adopt camera-controlled text-to-video generation as a testbed task. Within this setting, we further identify pitch and roll as two components effective for **Absolute Orientation Encoding**, enabling full control over the initial camera orientation. Together, these designs form **UCPE (Unified Camera Positional Encoding)**, which integrates into a pretrained video Diffusion Transformer through a lightweight spatial attention adapter, adding **less than 1% trainable parameters** while achieving state-of-the-art camera controllability and visual fidelity. To facilitate systematic training and evaluation, we construct a large video dataset covering a wide range of camera motions and lens types. Extensive experiments validate the effectiveness of UCPE in camera-controllable video generation and highlight its potential as a general camera representation for Transformers across future multi-view, video, and 3D tasks.