用线性上下文学习器从动态三维场景中提取像素特征
Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners
用现成网络估计的深度与运动线索图作为上下文，通过线性 in-context 学习从无标注视频中训练像素级、时序一致的特征描述子，可直接替换 VOS、表面法向估计、语义分割等密集预测任务中的骨干表征。
原文依据 The core element of our training framework is linear in-context learning.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉模型最令人兴奋的应用之一涉及像素级推理。尽管视觉基础模型已十分丰富，我们仍然缺乏能够在像素级别有效嵌入视觉场景时空属性的表征。现有框架要么在基于图像的预训练任务上训练，因而无法考虑动态元素；要么在视频序列上训练以进行动作级推理，因而无法扩展到密集像素级预测。我们提出了一个从视频中学习像素级精确特征描述子的框架 LILA。我们训练框架的核心要素是线性上下文学习。LILA 利用由现成网络估计得到的时空线索图——深度与运动。尽管这些线索本身带有噪声，LILA 仍能在未经筛选的视频数据集上有效训练，以时序一致的方式嵌入语义与几何属性。我们在多种视觉任务上展示了所学表征带来的显著经验收益：视频目标分割、表面法向估计与语义分割。
英文原文摘要 1091 字符
One of the most exciting applications of vision models involve pixel-level reasoning. Despite the abundance of vision foundation models, we still lack representations that effectively embed spatio-temporal properties of visual scenes at the pixel level. Existing frameworks either train on image-based pretext tasks, which do not account for dynamic elements, or on video sequences for action-level reasoning, which does not scale to dense pixel-level prediction. We present a framework that learns pixel-accurate feature descriptors from videos, LILA. The core element of our training framework is linear in-context learning. LILA leverages spatio-temporal cue maps--depth and motion--estimated with off-the-shelf networks. Despite the noisy nature of those cues, LILA trains effectively on uncurated video datasets, embedding semantic and geometric properties in a temporally consistent manner. We demonstrate compelling empirical benefits of the learned representation across a diverse suite of vision tasks: video object segmentation, surface normal estimation and semantic segmentation.