PanoEnv：在 panor 全景环境中用强化学习探索 3D 空间智能
PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning
针对 ERP 全景图的几何畸变与 3D 监督缺失，给出了一套可复用的方案：用带真值引导奖励的 GRPO 后训练，配合两阶段课程（先结构化任务、后开放式数据），把 3D 空间推理能力注入 VLM；做全景/全向感知或想给 VLM 补几何推理的可以直接借鉴这套奖励设计与课程安排。
原文依据 a reinforcement learning post-training framework based on Group Relative Policy Optimization (GRPO) with a ground-truth-guided reward
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论360 度全景图像正越来越多地用于 VR、自动驾驶和机器人领域，以实现整体场景理解。然而，当前视觉语言模型（VLM）在处理等距柱状投影（ERP）图像上的 3D 空间推理时，受几何畸变和 3D 监督有限的制约而表现不佳。我们提出 PanoEnv，一个由合成 3D 环境构建的大规模视觉问答基准，包含五大类别（如相对位置、属性比较）的 14.8K 个问题，并以精确的 3D 标注——深度、分割和边界框——作为依据。对 14 个最优 VLM 的基准评测显示其 3D 理解能力有限，总体准确率仅为 49.34%，开放式（OE）问题仅为 8.36%。为增强 3D 推理，我们提出一种基于组相对策略优化（GRPO）的强化学习后训练框架，其奖励由真值引导，融合了五种几何感知策略（如距离容差、空间一致性）。两阶段课程进一步缓解灾难性遗忘：第一阶段在结构化任务（判断题、多选题）上训练，第二阶段在混合开放式数据上微调以提升泛化能力。我们的 7B 模型创下新的最优性能，总体准确率提升至 52.93%（+3.59%），开放式准确率提升至 14.83%，同时保持结构化任务性能。它还取得最高的语义得分（Q-Score 6.24、P-Score 5.95），超越 32B 模型。这些结果表明，PanoEnv-QA 与我们的课程式强化学习框架能有效为 VLM 注入面向全向感知的 3D 空间智能。
英文原文摘要 1552 字符
360deg panoramic images are increasingly used in VR, autonomous driving, and robotics for holistic scene understanding. However, current Vision-Language Models (VLMs) struggle with 3D spatial reasoning on Equirectangular Projection (ERP) images due to geometric distortion and limited 3D supervision. We introduce PanoEnv, a large-scale VQA benchmark built from synthetic 3D environments, containing 14.8K questions across five categories (e.g., relative position, attribute comparison) grounded in accurate 3D annotations--depth, segmentation, and bounding boxes. Benchmarking 14 state-of-the-art VLMs reveals limited 3D understanding, achieving only 49.34% overall and 8.36% on open-ended (OE) questions. To enhance 3D reasoning, we propose a reinforcement learning post-training framework based on Group Relative Policy Optimization (GRPO) with a ground-truth-guided reward combining five geometry-aware strategies (e.g., distance tolerance, spatial consistency). A two-stage curriculum further mitigates catastrophic forgetting: Stage 1 trains on structured tasks (T/F, MCQ), and Stage 2 fine-tunes on mixed OE data for generalization. Our 7B model sets a new SoTA performance, improving total accuracy to 52.93% (+3.59%) and OE accuracy to 14.83% while maintaining structured-task performance. It also achieves top semantic scores (Q-Score 6.24, P-Score 5.95), surpassing 32B models. These results demonstrate that PanoEnv-QA and our curriculum-based RL framework effectively instill 3D spatial intelligence in VLMs for omnidirectional perception.