GaussianVision：利用2D高斯泼溅从压缩图像表示实现视觉-语言对齐
GaussianVision: Vision-Language Alignment from Compressed Image Representations using 2D Gaussian Splatting
用2D高斯泼溅作为紧凑、空间自适应的视觉表示替代RGB像素，配合冻结的RGB Transformer骨干和轻量输入茎，可显著降低传输与词元开销，为视觉语言模型提供可复用的高效视觉前端。
原文依据 We explore 2D Gaussian Splatting (2DGS) as an alternative visual substrate for alignment
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现代视觉-语言流水线由在海量图文语料上训练的RGB视觉编码器驱动。尽管这些流水线已实现令人印象深刻的零样本能力和跨任务的强迁移性，它们仍从像素域继承了两个结构性低效问题：(i) 从边缘设备向云端传输密集RGB图像能耗高且成本昂贵；(ii) 基于图块的词元化使序列长度爆炸，给注意力预算和上下文限制带来压力。我们探索2D高斯泼溅（2DGS）作为对齐的替代视觉基底：一种紧凑、空间自适应的表示，通过一组彩色各向异性高斯参数化图像。我们开发了可扩展的2DGS流水线，具有结构化初始化、亮度感知剪枝和批处理CUDA内核，相比先前实现实现了超过90倍的拟合加速和97%的GPU利用率。我们进一步通过复用冻结的基于RGB的Transformer骨干网络，并搭配轻量级泼溅感知输入茎和感知重采样器，将对比语言-图像预训练（CLIP）适配到2DGS，仅训练总参数的9.7-13.8%。在来自DataComp的1280万数据集上，GS编码器在CLIP基准的38个数据集上取得有竞争力的零样本性能，同时相对于像素将输入压缩3-23.5倍。当集成到完整的端到端LLaVA风格视觉-语言模型（VLM）中时，GS编码器在六个VQA基准上持续优于RGB VLM基线，且通常词元数更少，表明一种传输高效的结构化高斯表示可以在下游多模态推理中匹配或超越基于像素的模型。我们的代码库见https://github.com/Tambe-Lab/GaussianVision。
英文原文摘要 1750 字符
Modern vision-language pipelines are driven by RGB vision encoders trained on massive image-text corpora. While these pipelines have enabled impressive zero-shot capabilities and strong transfer across tasks, they still inherit two structural inefficiencies from the pixel domain: (i) transmitting dense RGB images from edge devices to the cloud is energy-intensive and costly, and (ii) patch-based tokenization explodes sequence length, stressing attention budgets and context limits. We explore 2D Gaussian Splatting (2DGS) as an alternative visual substrate for alignment: a compact, spatially adaptive representation that parameterizes images by a set of colored anisotropic Gaussians. We develop a scalable 2DGS pipeline with structured initialization, luminance-aware pruning, and batched CUDA kernels, achieving over 90x faster fitting and 97% GPU utilization compared to prior implementations. We further adapt contrastive language-image pre-training (CLIP) to 2DGS by reusing a frozen RGB-based transformer backbone with a lightweight splat-aware input stem and a perceiver resampler, training only 9.7 - 13.8% of the total parameters. On a 12.8M dataset from DataComp, GS encoders yield competitive zero-shot performance on 38 datasets from the CLIP benchmark while compressing inputs 3-23.5x relative to pixels. When integrated into a full end-to-end LLaVA-style visual-language model (VLM), GS encoders consistently outperform the RGB VLM baseline across six VQA benchmarks, often with reduced token counts, demonstrating that a transmission-efficient, structured Gaussian representation can match or surpass pixel-based models in downstream multimodal reasoning. Our codebase can be found at https://github.com/ Tambe-Lab/GaussianVision.