Merge3D：通过联合2D-3D词元合并实现高效3D多模态大语言模型
Merge3D: Efficient 3D Multimodal LLMs via Joint 2D-3D Token Merging
提出一种几何感知的词元合并框架，利用2D注意力选择语义显著词元，并结合2D+3D混合相似度聚合空间邻域上下文，可在激进压缩下保留3D结构先验与帧间对应关系，为多视图3D场景理解中的视觉词元冗余问题提供可复用方案。
原文依据 Merge3D bridges the modalities with a Semantic-Geometric Token Merger (SemGeo Merger): 2D attention is used to select semantically salient dominant tokens, while a hybrid 2D+3D similarity assigns and aggregates contextual tokens from spatially coherent 3D neighborhoods.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论融合3D几何信息的多模态大语言模型（MLLMs）在3D场景理解中展现出强大能力。然而，其主要瓶颈在于处理多视图、长视觉词元序列所带来的巨大计算负担。为应对这一挑战，我们提出Merge3D，一种几何感知的词元合并框架，同时整合3D几何与2D语义信息。仅依赖语义信号的传统2D压缩方法被证明不适用于3D任务，因为它们倾向于丢弃空间关键词元并损害视觉定位性能。Merge3D通过语义-几何词元合并器（SemGeo Merger）桥接两种模态：利用2D注意力选择语义显著的主导词元，同时采用混合2D+3D相似度从空间连贯的3D邻域中分配并聚合上下文词元。这在激进压缩下保留了3D结构先验与帧间对应关系。Merge3D实现了高达70%的视觉词元缩减和约3倍的推理加速，同时在Scan2Cap、CV-Bench和BLINK等3D视觉定位、描述生成和空间推理基准上保持强劲性能。
英文原文摘要 1198 字符
Multimodal Large Language Models (MLLMs) incorporating 3D geometry demonstrate significant power in 3D scene understanding. Their primary bottleneck, however, is the substantial computational burden associated with processing multi-view, lengthy visual token sequences. To surmount this challenge, we propose Merge3D, a geometry-aware token merging framework that integrates both 3D geometry and 2D semantic information. Conventional 2D compression methods, which rely solely on semantic signals, prove inadequate for 3D tasks, as they tend to discard spatially critical tokens and damage grounding performance. Merge3D bridges the modalities with a Semantic-Geometric Token Merger (SemGeo Merger): 2D attention is used to select semantically salient dominant tokens, while a hybrid 2D+3D similarity assigns and aggregates contextual tokens from spatially coherent 3D neighborhoods. This preserves 3D structural priors and inter-frame correspondences under aggressive compression. Merge3D achieves up to 70% visual token reduction and up to ~3xinference speedup, while retaining strong performance on 3D grounding, captioning, and spatial reasoning benchmarks such as Scan2Cap, CV-Bench, and BLINK.