图像-形状对应的最佳分割伙伴
Best Segmentation Buddies for Image-Shape Correspondence
提出将2D视觉模型特征蒸馏到3D形状表面，并通过最佳分割伙伴匹配建立图像分割与3D语义部件对应的方法，可复用于跨模态检索、三维部件分割和姿态迁移等任务。
原文依据 we first distill deep visual features from a 2D vision model onto the 3D shape surface
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论寻找对应关系是计算机视觉和图形学中一个基础且被广泛研究的问题。本文研究了在自然图像与无纹理三维形状之间估计分割到分割对应关系这一尚未充分探索的任务。由于外观、几何和视角的巨大差异，该任务极具挑战性。我们的方法通过将图像分割区域中的像素与三维形状对应语义部件中的顶点相链接，弥合了跨模态鸿沟。为此，我们首先将二维视觉模型的深层视觉特征蒸馏到三维形状表面，从而能够计算图像像素与形状顶点之间的特征相似度。然后，我们识别出最佳分割伙伴——即其最相似的图像像素位于图像分割区域内的顶点，从而可靠地发现语义对应形状部件中的顶点。最后，我们利用从二维图像分割模型蒸馏出的三维特征，直接在三维中对形状进行分割，以自举方式推进对应过程。我们在大量图像-形状对上展示了该方法的通用性和鲁棒性，呈现出准确且语义有意义的对应关系。项目页面见 https://threedle.github.io/bsb/。
英文原文摘要 1312 字符
Finding correspondences is a fundamental and extensively researched problem in computer vision and graphics. In this work, we examine the underexplored task of estimating segmentation-to-segmentation correspondence between images in the wild and untextured 3D shapes. This task is highly challenging due to substantial differences in appearance, geometry, and viewpoint. Our approach bridges the cross-modality gap by linking pixels in the image segment to vertices in the corresponding semantic part of the 3D shape. To achieve this, we first distill deep visual features from a 2D vision model onto the 3D shape surface, allowing for the computation of feature similarity between image pixels and shape vertices. Then, we identify Best Segmentation Buddies, vertices whose most similar image pixel lies within the image segmentation region, enabling the reliable discovery of vertices in semantically corresponding shape parts. Finally, we leverage distilled 3D features from the 2D image segmentation model to segment the shape directly in 3D, bootstrapping the correspondence process. We demonstrate the generality and robustness of our approach across a wide range of image-shape pairs, showcasing accurate and semantically meaningful correspondences. Our project page is at https://threedle.github.io/bsb/.