HAMMER：通过跨模态集成利用多模态大语言模型实现意图驱动的 3D 可供性定位
HAMMER: Harnessing MLLMs via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding
利用多模态大语言模型（MLLM）的语义推理能力，将参考图像中的交互意图编码为接触感知嵌入，并通过分层跨模态融合与多粒度几何提升实现 3D 可供性定位，为 3D 场景理解提供新思路。
原文依据 leverages emerging multimodal large language models (MLLMs) for interaction intention-driven 3D affordance grounding
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论人类通常通过观察图像或视频中的交互来推理物体的可供性，并且一旦形成这种知识，就可以泛化到新物体上。受此原理启发，我们提出了一种新颖的框架 HAMMER，利用新兴的多模态大语言模型（MLLMs）进行交互意图驱动的 3D 可供性定位。我们不是生成显式的物体属性描述或依赖现成的 2D 分割器，而是将参考图像中描绘的交互意图聚合为接触感知嵌入，并引导模型推断文本可供性标签，确保充分挖掘物体语义和上下文线索。我们进一步设计了一种分层跨模态集成机制，以充分利用 MLLM 的互补信息进行 3D 表示细化，并引入多粒度几何提升模块，将空间特性注入提取的意图嵌入中，从而促进准确的 3D 可供性定位。在公开数据集和我们新构建的损坏基准上的大量实验表明，与现有方法相比，我们的方法在可见和不可见场景中均具有优越性和鲁棒性。代码和模型将公开。
英文原文摘要 1327 字符
Humans commonly reason about object affordance through observed interactions in images or videos, and once formed, such knowledge can be generically generalized to novel objects. Inspired by this principle, we advocate for a novel framework that leverages emerging multimodal large language models (MLLMs) for interaction intention-driven 3D affordance grounding, namely HAMMER. Instead of generating explicit object attribute descriptions or relying on off-the-shelf 2D segmenters, we alternatively aggregate the interaction intention depicted in the reference image into a contact-aware embedding and guide the model to infer textual affordance labels, ensuring it thoroughly excavates object semantics and contextual cues. We further devise a hierarchical cross-modal integration mechanism to fully exploit the complementary information from the MLLM for 3D representation refinement and introduce a multi-granular geometry lifting module that infuses spatial characteristics into the extracted intention embedding, thus facilitating accurate 3D affordance localization. Extensive experiments on public datasets and our newly constructed corrupted benchmark reveal the superiority and robustness of our method in seen and unseen scenarios compared to existing approaches. The code and models will be made publicly available.