OpenVision 2：面向多模态学习的一族生成式预训练视觉编码器
OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning
去掉文本编码器和对比损失、仅用captioning loss训练视觉编码器，可作为一种更省显存和时间的视觉表征预训练方案，直接用于多模态大模型的视觉塔初始化。
原文依据 we remove the text encoder (and therefore the contrastive loss), retaining only the captioning loss as a purely generative training signal
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文对 OpenVision 的架构与损失设计进行了简化，以提升其训练效率。遵循此前的视觉-语言预训练工作 CapPa 与 AIMv2，以及 LLaVA 等现代多模态设计，我们的改动直接明了：移除文本编码器（因而移除对比损失），仅保留描述生成损失作为纯粹的生成式训练信号。我们将这一新版本命名为 OpenVision 2。初步结果令人鼓舞：尽管进行了上述简化，OpenVision 2 在广泛的多模态基准上仍能与原模型性能相当，同时大幅削减训练时间与内存消耗。例如，在 ViT-L/14 上，它将训练时间缩短约 1.5 倍（从 83 小时降至 57 小时），内存占用降低约 1.8 倍（从 24.5GB 降至 13.8GB，等效于最大批大小从 2k 增至 8k）。这种优越的训练效率还使我们能够远超 OpenVision 中所用最大视觉编码器的规模，达到超过 10 亿参数。我们坚信，这种轻量化的纯生成式范式对未来多模态基础模型中的视觉编码器发展具有吸引力。
英文原文摘要 1219 字符
This paper provides a simplification on OpenVision's architecture and loss design for enhancing its training efficiency. Following the prior vision-language pretraining works CapPa and AIMv2, as well as modern multimodal designs like LLaVA, our changes are straightforward: we remove the text encoder (and therefore the contrastive loss), retaining only the captioning loss as a purely generative training signal. We name this new version OpenVision 2. The initial results are promising: despite this simplification, OpenVision 2 competitively matches the original model's performance on a broad set of multimodal benchmarks while substantially cutting both training time and memory consumption. For example, with ViT-L/14, it reduces training time by about 1.5x (from 83h to 57h), and memory usage by about 1.8x (from 24.5GB to 13.8GB, equivalently allowing the maximum batch size to grow from 2k to 8k). This superior training efficiency also allows us to scale far beyond the largest vision encoder used in OpenVision, reaching more than 1 billion parameters. We hold a strong belief that this lightweight, generative-only paradigm is compelling for future vision encoder development in multimodal foundation models.