大规模编解码化身:大规模化身预训练的惊人有效性
Large-scale Codec Avatars: The Unreasonable Effectiveness of Large-scale Avatar Pretraining
提出首个面向3D虚拟化身的大规模预训练/后训练范式,利用100万野外视频学习外观与几何先验,再以高质量数据微调,实现前馈式全身化身生成,对视觉研究者而言是可复用的『大规模预训练+高质量微调』策略在3D生成与重建中的范例,并展示了零样本鲁棒性与可重光照等涌现能力。
原文依据 we present, for the first time, a pre/post-training paradigm for 3D avatar modeling at scale
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论高质量3D化身建模面临保真度与泛化性之间的关键权衡。一方面,多视角影棚数据能够对人体进行高保真建模,并精确控制表情与姿态,但由于规模有限以及影棚环境与真实世界之间的域差距,难以泛化到真实数据。另一方面,近期基于数百万野外样本训练的大规模化身模型展现出在广泛身份上泛化的潜力,但由于固有的3D歧义,生成的化身往往质量较低。为解决这一问题,我们提出大规模编解码化身(LCA),一种高保真、全身3D化身模型,能够以前馈方式泛化到世界规模的人群,并实现高效推理。受大语言模型和视觉基础模型成功的启发,我们首次提出面向大规模3D化身建模的预训练/后训练范式:在100万野外视频上预训练以学习外观与几何的广泛先验,然后在高质量精选数据上后训练以增强表现力与保真度。LCA能够泛化到不同发型、服装和人口特征,同时提供精确、细粒度的面部表情与手指级关节控制,并具有强身份保持能力。值得注意的是,我们观察到对可重光照性和宽松服装支持在无约束输入上的涌现泛化,以及对风格化图像的零样本鲁棒性,尽管缺乏直接监督。
英文原文摘要 1555 字符
High-quality 3D avatar modeling faces a critical trade-off between fidelity and generalization. On the one hand, multi-view studio data enables high-fidelity modeling of humans with precise control over expressions and poses, but it struggles to generalize to real-world data due to limited scale and the domain gap between the studio environment and the real world. On the other hand, recent large-scale avatar models trained on millions of in-the-wild samples show promise for generalization across a wide range of identities, yet the resulting avatars are often of low-quality due to inherent 3D ambiguities. To address this, we present Large-Scale Codec Avatars (LCA), a high-fidelity, full-body 3D avatar model that generalizes to world-scale populations in a feedforward manner, enabling efficient inference. Inspired by the success of large language models and vision foundation models, we present, for the first time, a pre/post-training paradigm for 3D avatar modeling at scale: we pretrain on 1M in-the-wild videos to learn broad priors over appearance and geometry, then post-train on high-quality curated data to enhance expressivity and fidelity. LCA generalizes across hair styles, clothing, and demographics while providing precise, fine-grained facial expressions and finger-level articulation control, with strong identity preservation. Notably, we observe emergent generalization to relightability and loose garment support to unconstrained inputs, and zero-shot robustness to stylized imagery, despite the absence of direct supervision.