Proxy3D：通过语义聚类与对齐为视觉语言模型构建高效三维表征
Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment
用语义感知聚类把视频帧特征压缩成一组 3D 代理 token 替代像素对齐的视觉序列，为 VLM 提供紧凑且具空间一致性的视觉输入；做 3D 空间推理或多模态大模型视觉编码器压缩时可复用这一「聚类成代理 + 多阶段对齐」范式。
原文依据 we employ semantic and geometric encoders to extract scene features and then perform their semantic-aware clustering to obtain a set of proxies in the 3D space
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）中的空间智能因其在三维世界中进行推理的实际需求而受到研究关注。尽管已有结果令人期待，大多数现有方法仍沿用 VLM 中传统的二维流程，为视觉模态使用像素对齐的表征。然而，基于对应关系的模型虽具有隐式三维场景理解，却往往无法实现空间一致性；而基于表征的模型虽具备三维几何先验，却在视觉序列的序列化上缺乏效率。为解决这一问题，我们提出 Proxy3D 方法，为视觉模态构建紧凑而全面的三维代理表征。仅以视频帧作为输入，我们采用语义编码器与几何编码器提取场景特征，随后对其进行语义感知聚类，从而在三维空间中获得一组代理。为进行表征对齐，我们进一步构建了 SpaceSpan 数据集，并采用多阶段训练，使所提出的三维代理表征与 VLM 相适配。在使用更短的视觉信息序列时，我们的方法在三维视觉问答、视觉定位以及通用空间智能基准上取得了有竞争力或最优的性能。
英文原文摘要 1208 字符
Spatial intelligence in vision-language models (VLMs) attracts research interest with the practical demand to reason in the 3D world. Despite promising results, most existing methods follow the conventional 2D pipeline in VLMs and use pixel-aligned representations for the vision modality. However, correspondence-based models with implicit 3D scene understanding often fail to achieve spatial consistency, and representation-based models with 3D geometric priors lack efficiency in vision sequence serialization. To address this, we propose a Proxy3D method with compact yet comprehensive 3D proxy representations for the vision modality. Given only video frames as input, we employ semantic and geometric encoders to extract scene features and then perform their semantic-aware clustering to obtain a set of proxies in the 3D space. For representation alignment, we further curate the SpaceSpan dataset and apply multi-stage training to adopt the proposed 3D proxy representations with the VLM. When using shorter sequences for vision information, our method achieves competitive or state-of-the-art performance in 3D visual question answering, visual grounding and general spatial intelligence benchmarks.