Co-Me:面向视觉几何 Transformer 的置信度引导词元合并
Co-Me: Confidence Guided Token Merging for Visual Geometric Transformers
对做多视图/流式几何 Transformer(VGGT、Pi3)的人,这是一种即插即用的免训练加速手段:用轻量置信度预测器给词元排序并合并低置信词元,在保持空间覆盖的同时把序列长度带来的算力开销压下来,可直接用于实时 3D 感知与重建的部署。
原文依据 distilled a light-weight confidence predictor to rank tokens by uncertainty and selectively merge low-confidence ones
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出置信度引导词元合并(Co-Me),一种无需重新训练或微调基座模型的视觉几何 Transformer 加速机制。Co-Me 蒸馏出一个轻量级置信度预测器,按不确定性对词元排序,并选择性地合并低置信度词元,从而在保持空间覆盖的同时有效降低计算量。与基于相似度的合并或剪枝相比,Co-Me 中的置信度信号能够可靠地指示 Transformer 所强调的区域,从而在不损失性能的前提下实现显著加速。Co-Me 可无缝应用于多种多视图与流式视觉几何 Transformer,其加速比随序列长度增长而扩展。应用于 VGGT 与 Pi3 时,Co-Me 分别实现最高 21.5 倍与 20.4 倍的加速,使视觉几何 Transformer 可用于实时三维感知与重建。
英文原文摘要 872 字符
We propose Confidence-Guided Token Merging (Co-Me), an acceleration mechanism for visual geometric transformers without retraining or finetuning the base model. Co-Me distilled a light-weight confidence predictor to rank tokens by uncertainty and selectively merge low-confidence ones, effectively reducing computation while maintaining spatial coverage. Compared to similarity-based merging or pruning, the confidence signal in Co-Me reliably indicates regions emphasized by the transformer, enabling substantial acceleration without degrading performance. Co-Me applies seamlessly to various multi-view and streaming visual geometric transformers, achieving speedups that scale with sequence length. When applied to VGGT and Pi3, Co-Me achieves up to 21.5x and 20.4x speedup, making visual geometric transformers practical for real-time 3D perception and reconstruction.