视频基础模型编码了多少 3D 信息？
How Much 3D Do Video Foundation Models Encode?
提出一种模型无关的探针框架，通过浅层读出从视频基础模型特征中估计多种 3D 属性，为评估视觉模型内在 3D 理解提供可复用工具，并揭示视频生成模型可超越专家 3D 模型。
原文依据 We propose the first model-agnostic framework that measures the 3D awareness of various VidFMs by estimating multiple 3D properties from their features via shallow read-outs.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频是三维世界的连续二维投影。在大规模视频数据上训练后，全局 3D 理解会自然涌现吗？我们通过量化现有视频基础模型（VidFMs）的 3D 理解来研究这一问题，这些模型在海量视频数据上预训练。我们提出了首个模型无关的框架，通过浅层读出从视频基础模型的特征中估计多种 3D 属性，从而衡量各种视频基础模型的 3D 感知能力。我们的研究在多个维度上呈现了关于视频基础模型 3D 感知的有意义发现。特别地，我们表明最优的视频生成模型对 3D 物体和场景表现出强大的理解能力，尽管它们未在任何 3D 数据上训练。这种理解甚至能超越专门为 3D 任务训练的大型专家模型。我们的发现以及对主要视频基础模型的 3D 基准测试，为构建可扩展的 3D 模型提供了有价值的观察。
英文原文摘要 923 字符
Videos are continuous 2D projections of 3D worlds. After training on large video data, will global 3D understanding naturally emerge? We study this by quantifying the 3D understanding of existing Video Foundation Models (VidFMs) pretrained on vast video data. We propose the first model-agnostic framework that measures the 3D awareness of various VidFMs by estimating multiple 3D properties from their features via shallow read-outs. Our study presents meaningful findings regarding the 3D awareness of VidFMs on multiple axes. In particular, we show that state-of-the-art video generation models exhibit a strong understanding of 3D objects and scenes, despite not being trained on any 3D data. Such understanding can even surpass that of large expert models specifically trained for 3D tasks. Our findings, together with the 3D benchmarking of major VidFMs, provide valuable observations for building scalable 3D models.