基于多视图扩散先验的无监督单目三维关键点发现
Unsupervised Monocular 3D Keypoint Discovery from Multi-View Diffusion Priors
把预训练多视图扩散模型当作几何先验来源：用单图生成的多视图图像作为监督信号，并把扩散特征转成显式三维特征体，从而在无标注、无标定多视图的条件下做单目三维关键点估计；这套「扩散先验→三维监督」的流程可迁移到其他缺乏三维标注的物体/人体结构化任务。
原文依据 we leverage powerful geometric priors embedded in a pretrained multi-view diffusion model
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有大多数三维关键点估计方法依赖人工标注或标定的多视图图像，而这两类数据的采集成本都很高。本文提出 KeyDiff3D，一个能够从单张图像准确预测三维关键点的框架，从而免除了对上述昂贵数据采集的需求。为实现这一目标，我们利用了预训练多视图扩散模型中蕴含的强大几何先验。在我们的框架中，扩散模型由单张图像生成多视图图像，这些图像作为监督信号为模型提供三维几何线索。我们还引入了一个三维特征提取器，将扩散特征中隐含的三维先验转换为显式的三维特征体。除准确的关键点估计之外，我们进一步提出了一套流程，使得能够对扩散模型生成的三维物体进行操控。在包括 Human3.6M、CUB-200-2011、Stanford Dogs 以及若干野外与域外输入在内的多种数据集上的实验结果，凸显了本方法在准确性、泛化性以及支持从单张图像操控扩散模型所生成三维物体方面的有效性。
英文原文摘要 1195 字符
Most existing 3D keypoint estimation methods rely on manual annotations or calibrated multi-view images, both of which are expensive to collect.This paper introduces KeyDiff3D, a framework that can accurately predict 3D keypoints from a single image, thus eliminating the need for such expensive data acquisitions.To achieve this, we leverage powerful geometric priors embedded in a pretrained multi-view diffusion model.In our framework, the diffusion model generates multi-view images from a single image, serving as supervision signals to provide 3D geometric cues to our model.We also introduce a 3D feature extractor that transforms implicit 3D priors embedded in the diffusion features into explicit 3D feature volumes.Beyond accurate keypoint estimation, we further introduce a pipeline that enables manipulation of 3D objects generated by the diffusion model.Experimental results on diverse datasets, including Human3.6M, CUB-200-2011, Stanford Dogs, and several in-the-wild and out-of-domain inputs, highlight the effectiveness of our method in terms of accuracy, generalization, and its ability to enable manipulation of 3D objects generated by the diffusion model from a single image.