VideoWeaver：面向具身智能体的多模态多视图视频到视频翻译
VideoWeaver: Multimodal Multi-View Video-to-Video Transfer for Embodied Agents
将多视图视频翻译统一到基于前馈空间基础模型（Pi3）的共享4D潜空间，并用不同扩散时间步训练实现视图一致的自回归新视角合成，为具身智能中的世界随机化提供可复用方案。
原文依据 we propose to ground all views in a shared 4D latent space derived from a feed-forward spatial foundation model, namely, Pi3.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期视频到视频（V2V）翻译的进展使得具身AI演示的真实感重仿真成为可能，这一能力允许预训练的机器人策略无需额外数据收集即可迁移到新环境。然而，先前的工作一次只能处理单个视图，而具身AI任务通常由多个同步相机捕获以支持策略学习。独立地将单视图模型朴素地应用于每个相机会导致跨视图外观不一致，且标准Transformer架构由于跨视图注意力的二次成本而无法扩展到多视图设置。我们提出VideoWeaver，首个多模态多视图V2V翻译框架。VideoWeaver最初被训练为单视图基于流的V2V模型。为了实现向多视图范式的扩展，我们提出将所有视图锚定在一个共享的4D潜空间中，该潜空间源自前馈空间基础模型Pi3。这鼓励即使在宽基线和动态相机运动下也保持视图一致的外观。为了扩展到固定数量的相机之外，我们在不同的扩散时间步上训练视图，使模型能够学习联合和条件视图分布。这进而允许以现有视图为条件自回归地合成新视角。实验表明，在单视图翻译基准上，其性能优于或类似于最优方法，并首次实现了物理和风格一致的多视图翻译，包括对机器人学习中的世界随机化至关重要的具有挑战性的自我中心和异构相机设置。
英文原文摘要 1663 字符
Recent progress in video-to-video (V2V) translation has enabled realistic resimulation of embodied AI demonstrations, a capability that allows pretrained robot policies to be transferable to new environments without additional data collection. However, prior works can only operate on a single view at a time, while embodied AI tasks are commonly captured from multiple synchronized cameras to support policy learning. Naively applying single-view models independently to each camera leads to inconsistent appearance across views, and standard transformer architectures do not scale to multi-view settings due to the quadratic cost of cross-view attention. We present VideoWeaver, the first multimodal multi-view V2V translation framework. VideoWeaver is initially trained as a single-view flow-based V2V model. To achieve an extension to the multi-view regime, we propose to ground all views in a shared 4D latent space derived from a feed-forward spatial foundation model, namely, Pi3. This encourages view-consistent appearance even under wide baselines and dynamic camera motion. To scale beyond a fixed number of cameras, we train views at distinct diffusion timesteps, enabling the model to learn both joint and conditional view distributions. This in turn allows autoregressive synthesis of new viewpoints conditioned on existing ones. Experiments show superior or similar performance to the state-of-the-art on the single-view translation benchmarks and, for the first time, physically and stylistically consistent multi-view translations, including challenging egocentric and heterogeneous-camera setups central to world randomization for robot learning.