GeoSAM2:释放 SAM2 在三维部件分割中的能力
GeoSAM2: Unleashing the Power of SAM2 for 3D Part Segmentation
把 3D 部件分割转成多视图 2D 掩码预测,用 SAM2 加低秩适配和残差几何融合,接受点击/框提示,再把掩码反投影聚合到物体上;做 3D 分割或想复用 SAM2 先验的人可直接借鉴这套提示可控、免逐形状优化的流程。
原文依据 casts the task as multi-view 2D mask prediction
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出 GeoSAM2,一个提示可控的三维部件分割框架,将任务转化为多视图二维掩码预测。给定一个无纹理物体,我们从预定义视角渲染法线图与点图,并接受简单的二维提示——点击或框——来引导部件选择。这些提示由共享的 SAM2 骨干网络处理,该骨干网络经低秩适配与残差几何融合增强,从而在保留预训练先验的同时实现视角特定的推理。预测的掩码被反投影到物体上,并在各视图间聚合。我们的方法无需文本提示、逐形状优化或完整三维标注,即可实现细粒度、部件特定的控制。与全局聚类或基于尺度的方法相比,提示是显式的、空间上有依据的且可解释的。我们在 PartObjaverse-Tiny 与 PartNetE 上取得最优的类别无关性能,优于缓慢的基于优化的流程以及快速但粗糙的前馈方法。我们的结果凸显了一种新范式:使三维分割的范式与 SAM2 对齐,利用交互式二维输入释放物体级部件理解中的可控性与精度。
英文原文摘要 1177 字符
We introduce GeoSAM2, a prompt-controllable framework for 3D part segmentation that casts the task as multi-view 2D mask prediction. Given a textureless object, we render normal and point maps from predefined viewpoints and accept simple 2D prompts--clicks or boxes--to guide part selection. These prompts are processed by a shared SAM2 backbone augmented with LoRA and residual geometry fusion, enabling view-specific reasoning while preserving pretrained priors. The predicted masks are back-projected to the object, aggregated across views.Our method enables fine-grained, part-specific control without requiring text prompts, per-shape optimization, or full 3D labels. In contrast to global clustering or scale-based methods, prompts are explicit, spatially grounded, and interpretable. We achieve state-of-the-art class-agnostic performance on PartObjaverse-Tiny and PartNetE, outperforming both slow optimization-based pipelines and fast but coarse feedforward approaches. Our results highlight a new paradigm: aligning the paradigm of 3D segmentation with SAM2, leveraging interactive 2D inputs to unlock controllability and precision in object-level part understanding.