以2D语义知识解锁3D功能可供性分割
Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge
利用2D视觉基础模型的语义知识，通过跨模态亲和力迁移预训练3D编码器，为点云功能可供性分割提供结构化特征学习方案。
原文依据 we leverage the rich semantic knowledge embedded in large-scale 2D Vision Foundation Models (VFMs) to guide 3D representation learning through a cross-modal alignment mechanism.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论功能可供性分割旨在将3D物体分解为承担不同功能角色的部件，使模型能够推理物体交互而非仅仅识别。现有方法大多遵循3D语义分割或基于提示的框架范式，当几何线索微弱或模糊时表现不佳，因为稀疏点云提供的功能信息有限。为克服这一局限，我们利用大规模2D视觉基础模型（VFM）中嵌入的丰富语义知识，通过跨模态对齐机制引导3D表征学习。具体而言，我们提出跨模态亲和力迁移（CMAT），这是一种预训练策略，迫使3D编码器与由提升的2D特征所诱导的语义结构对齐。CMAT由核心的亲和力对齐目标驱动，并辅以两个辅助损失——几何重建和特征多样性，共同促进结构化和判别性特征学习。在CMAT预训练的主干网络基础上，我们采用轻量级功能可供性分割器，通过高效的交叉注意力接口将文本或视觉提示注入学习到的3D空间，实现密集且提示感知的功能可供性预测，同时保持预训练期间建立的语义组织。大量实验表明，在准确性和效率上均持续优于先前的最优方法。
英文原文摘要 1458 字符
Affordance segmentation aims to decompose 3D objects into parts that serve distinct functional roles, enabling models to reason about object interactions rather than mere recognition. Existing methods, mostly following the paradigm of 3D semantic segmentation or prompt-based frameworks, struggle when geometric cues are weak or ambiguous, as sparse point clouds provide limited functional information. To overcome this limitation, we leverage the rich semantic knowledge embedded in large-scale 2D Vision Foundation Models (VFMs) to guide 3D representation learning through a cross-modal alignment mechanism. Specifically, we propose Cross-Modal Affinity Transfer (CMAT), a pretraining strategy that compels the 3D encoder to align with the semantic structures induced by lifted 2D features. CMAT is driven by a core affinity alignment objective, supported by two auxiliary losses, geometric reconstruction and feature diversity, which together encourage structured and discriminative feature learning. Built upon the CMAT-pretrained backbone, we employ a lightweight affordance segmentor that injects text or visual prompts into the learned 3D space through an efficient cross-attention interface, enabling dense and prompt-aware affordance prediction while preserving the semantic organization established during pretraining. Extensive experiments demonstrate consistent improvements over previous state-of-the-art methods in both accuracy and efficiency.