RealVLG-R1：面向机器人感知与操作的大规模真实世界视觉-语言定位基准
RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation
该工作提供了带边界框、分割掩码、抓取位姿与接触点统一标注的大规模真实场景视觉-语言定位数据集，并用强化微调让 VLM 在统一输出头下同时预测定位与抓取参数，可作为语言驱动操作与开放词表定位的联合评测基准。
原文依据 RealVLG-11B dataset provides multi-granularity annotations including bounding boxes, segmentation masks, grasp poses, contact points
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉-语言定位旨在建立自然语言与视觉实体之间的语义对应关系，使模型能够根据文本指令准确识别并定位目标物体。现有视觉-语言定位方法聚焦于粗粒度的物体级定位，而传统机器人抓取方法主要依赖几何线索、缺乏语言引导，这限制了它们在语言驱动操作场景中的适用性。为解决上述局限，我们提出 RealVLG 框架，将 RealVLG-11B 数据集与 RealVLG-R1 模型相结合，统一真实世界中的视觉-语言定位与抓取任务。RealVLG-11B 数据集提供多粒度标注，包括边界框、分割掩码、抓取位姿、接触点以及经人工核验的细粒度语言描述，覆盖约 165,000 张图像、800 余个物体实例、130 万条分割、检测与语言标注，以及约 110 亿个抓取样例。在该数据集基础上，RealVLG-R1 对预训练大规模视觉-语言模型进行强化微调，使其在给定自然语言指令时以统一方式预测边界框、分割掩码、抓取位姿与接触点。实验结果表明，RealVLG 支持在真实世界未见环境中的零样本感知与操作，建立了一个统一的语义-视觉多模态基准，为语言驱动的机器人感知与抓取策略学习提供了完整的数据与评测平台。全部数据与代码已公开于 https://github.com/lif314/RealVLG-R1。
英文原文摘要 1682 字符
Visual-language grounding aims to establish semantic correspondences between natural language and visual entities, enabling models to accurately identify and localize target objects based on textual instructions. Existing VLG approaches focus on coarse-grained, object-level localization, while traditional robotic grasping methods rely predominantly on geometric cues and lack language guidance, which limits their applicability in language-driven manipulation scenarios. To address these limitations, we propose the RealVLG framework, which integrates the RealVLG-11B dataset and the RealVLG-R1 model to unify real-world visual-language grounding and grasping tasks. RealVLG-11B dataset provides multi-granularity annotations including bounding boxes, segmentation masks, grasp poses, contact points, and human-verified fine-grained language descriptions, covering approximately 165,000 images, over 800 object instances, 1.3 million segmentation, detection, and language annotations, and roughly 11 billion grasping examples. Building on this dataset, RealVLG-R1 employs Reinforcement Fine-tuning on pretrained large-scale vision-language models to predict bounding boxes, segmentation masks, grasp poses, and contact points in a unified manner given natural language instructions. Experimental results demonstrate that RealVLG supports zero-shot perception and manipulation in real-world unseen environments, establishing a unified semantic-visual multimodal benchmark that provides a comprehensive data and evaluation platform for language-driven robotic perception and grasping policy learning. All data and code are publicly available at https://github.com/lif314/RealVLG-R1.