PointAlign：面向3D视觉语言模型的特征级对齐正则化
PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models
提出一种特征级对齐正则化方法，通过一致性损失约束LLM中间点云token与视觉输入token对齐，仅训练轻量投影器和LoRA适配器，可复用于其他多模态大模型中以保留几何信息。
原文依据 we constrain the intermediate point cloud tokens within the LLM to align with visual input tokens via a consistency loss
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论3D视觉语言模型（VLMs）在机器人、自动驾驶和增强现实等应用中至关重要，但其发展严重受限于配对3D-文本数据的稀缺。现有方法仅依赖下一词预测损失，仅使用语言token进行监督，导致有限3D数据的利用效率低下，并在中间表示中造成宝贵几何信息的显著退化与丢失。为解决这些局限，我们提出PointAlign，一种新颖的特征级对齐正则化方法。PointAlign显式监督中间点云表示，以在整个语言建模过程中保留细粒度的3D几何-语义信息。具体而言，我们通过一致性损失约束LLM内部的中间点云token与视觉输入token对齐。仅训练轻量对齐投影器和LoRA适配器，PointAlign以最小计算开销实现显式的特征级监督，有效防止几何退化。在ModelNet40和Objaverse数据集上的大量实验表明，我们的方法在分类任务上平均提升2.08个百分点，在具有挑战性的开放词表Objaverse分类任务上大幅提升7.50个百分点，在由Qwen2-72B-Instruct评估的3D物体描述生成任务上提升4.88个百分点，验证了PointAlign的有效性。
英文原文摘要 1461 字符
The development of 3D Vision-Language Models (VLMs), crucial for applications in robotics, autonomous driving, and augmented reality, is severely constrained by the scarcity of paired 3D-text data. Existing methods rely solely on next-token prediction loss, using only language tokens for supervision. This results in inefficient utilization of limited 3D data and leads to a significant degradation and loss of valuable geometric information in intermediate representations.To address these limitations, we propose \mname , a novel feature-level alignment regularization method. \mname explicitly supervises intermediate point cloud representations to preserve fine-grained 3D geometric-semantic information throughout the language modeling process. Specifically, we constrain the intermediate point cloud tokens within the LLM to align with visual input tokens via a consistency loss. By training only a lightweight alignment projector and LoRA adapters, \mname achieves explicit feature-level supervision with minimal computational overhead, effectively preventing geometric degradation.Extensive experiments on ModelNet40 and Objaverse datasets demonstrate that our method achieves 2.08 pp improvement on average for classification tasks, with a substantial 7.50 pp gain on the challenging open-vocabulary Objaverse classification task and 4.88 pp improvement on 3D object captioning evaluated by Qwen2-72B-Instruct, validating the effectiveness of \mname .