Molmo2：面向视频理解与视觉定位的视觉语言模型开放权重与数据
Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
Molmo2 提供了不依赖闭源 VLM 生成的视频/多图数据集与训练配方（packing 与 message-tree 编码、视觉 token 双向注意力、token 加权策略），并给出点驱动视觉定位与像素级跟踪的评测设定，可直接用于构建或评测具备指点和跟踪能力的视频语言模型。
原文依据 we show bi-directional attention on vision tokens and a novel token-weight strategy improves performance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当前最强的视频语言模型（VLM）仍为闭源，而最强的开放权重模型往往依赖专有 VLM 生成的合成数据，且不公开其训练数据或训练配方。因此，开源社区缺乏改进最优视频（及图像）语言模型所需的基础。关键在于，许多下游应用需要视觉定位——或通过指点、或通过像素级跟踪来实现。即便是专有模型也缺乏这一能力。我们提出 Molmo2，这是一个新的 VLM 系列，在开源模型中达到最优水平，并在单图像、多图像与视频任务中的点驱动视觉定位方面展现出卓越的新能力。我们的核心贡献是一组 7 个新视频数据集与 2 个多图像数据集，包括用于预训练的高细节视频描述数据集、用于微调的自由形式视频问答数据集、一个带有复杂查询的新目标跟踪数据集，以及一个创新的新视频指点数据集，全部在未使用闭源 VLM 的情况下采集。我们还给出了针对这些数据的训练配方，采用高效的打包与消息树编码方案，并表明视觉 token 上的双向注意力与一种新的 token 加权策略可提升性能。我们最优的 8B 模型在短时视频、计数与描述生成上优于开放权重与数据模型中的同类模型，在长视频上亦具竞争力。在视频视觉定位方面，Molmo2 显著优于 Qwen3-VL 等现有开放权重模型（视频计数准确率 35.5 对 29.6），并在部分任务上超越 Gemini 3 Pro 等专有模型（视频指点 F1 为 38.4 对 20.0，视频跟踪 J&F 为 56.2 对 41.1）。我们的模型权重、新数据集与源代码见 https://allenai.org/blog/molmo2。
英文原文摘要 1818 字符
Today's strongest video-language models (VLMs) remain proprietary, and the strongest open-weight models often rely on synthetic data from proprietary VLMs and do not disclose their training data or recipe. As a result, the open-source community lacks the foundations needed to improve on the state-of-the-art video (and image) language models. Crucially, many downstream applications require grounding--either by pointing or by tracking in pixels. Even proprietary models lack this capability. We present Molmo2, a new family of VLMs that are state-of-the-art among open-source models and demonstrate exceptional new capabilities in point-driven grounding in single image, multi-image, and video tasks. Our key contribution is a collection of 7 new video datasets and 2 multi-image datasets, including a dataset of highly detailed video captions for pre-training, a free-form video Q&A dataset for fine-tuning, a new object tracking dataset with complex queries, and an innovative new video pointing dataset, all collected without the use of closed VLMs. We also present a training recipe for this data utilizing an efficient packing and message-tree encoding scheme, and show bi-directional attention on vision tokens and a novel token-weight strategy improves performance. Our best-in-class 8B model outperforms others in the class of open weight and data models on short videos, counting, and captioning, and is competitive on long-videos. On video-grounding Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on video counting) and surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on video pointing and 56.2 vs 41.1 J&F on video tracking). Our model weights, new datasets, and source code are available at https://allenai.org/blog/molmo2.