高效无编码器的基于傅里叶的 3D 大型多模态模型
Efficient Encoder-Free Fourier-based 3D Large Multimodal Model
用空间填充曲线序列化加 FFT 近似自注意力，替代重型 3D 编码器做全局上下文建模，并用傅里叶增强的低秩适配把频域交互注入 LLM，为视觉侧设计轻量 3D 词元化与无编码器多模态架构提供可复用范式。
原文依据 a novel tokenizer that combines point cloud serialization and the Fast Fourier Transform (FFT) to approximate self-attention
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论处理 3D 数据的大型多模态模型（LMM）通常依赖沉重的预训练视觉编码器来提取几何特征。尽管近期的 2D 大型多模态模型已开始为提升效率与可扩展性而去除这类编码器，但将该范式扩展到 3D 仍具挑战，原因在于点云的无序性与大规模特性。这留下了一个关键且尚未回答的问题：如何设计一种大型多模态模型，在没有笨重编码器的情况下高效且有效地对无序 3D 数据进行词元化？我们提出 Fase3D，首个高效的无编码器、基于傅里叶的 3D 场景大型多模态模型。Fase3D 通过一种新颖的分词器应对可扩展性与置换不变性的挑战，该分词器结合点云序列化与快速傅里叶变换（FFT）来近似自注意力。这一设计支撑起一个有效且计算开销极小的架构，其建立在三项关键创新之上：第一，我们通过结构化超点紧凑地表示大场景。第二，我们的空间填充曲线序列化及随后的 FFT 实现了高效的全局上下文建模与基于图的词元合并。最后，我们的傅里叶增强低秩适配器以可忽略的成本向 LLM 主干注入全局频率感知交互。Fase3D 取得了与基于编码器的 3D 大型多模态模型相当的性能，同时在计算与参数量上显著更为高效。
英文原文摘要 1361 字符
Large Multimodal Models (LMMs) that process 3D data typically rely on heavy, pretrained visual encoders to extract geometric features. While recent 2D LMMs have begun to eliminate such encoders for efficiency and scalability, extending this paradigm to 3D remains challenging due to the unordered and large-scale nature of point clouds. This leaves a critical unanswered question: How can we design an LMM that tokenizes unordered 3D data effectively and efficiently without a cumbersome encoder? We propose Fase3D, the first efficient encoder-free Fourier-based 3D scene LMM. Fase3D tackles the challenges of scalability and permutation invariance with a novel tokenizer that combines point cloud serialization and the Fast Fourier Transform (FFT) to approximate self-attention. This design enables an effective and computationally minimal architecture, built upon three key innovations: First, we represent large scenes compactly via structured superpoints. Second, our space-filling curve serialization followed by an FFT enables efficient global context modeling and graph-based token merging. Lastly, our Fourier-augmented LoRA adapters inject global frequency-aware interactions into LLM backbones at a negligible cost. Fase3D achieves performance comparable to encoder-based 3D LMMs while being significantly more efficient in computation and parameters.