超越简单相加：面向恶劣全景场景的全景-语言模型
More than the Sum: Panorama-Language Models for Adverse Omni-Scenes
提出即插即用的全景稀疏注意力模块，让已有的针孔式 VLM 无需重训练即可处理等距柱状全景图，为 360° 视觉语言推理提供了可复用的适配手段；PanoVQA 数据集则提供了遮挡与事故等恶劣全景场景下的评测基准。
原文依据 a plug-and-play panoramic sparse attention module that allows existing pinhole-based VLMs to process equirectangular panoramas without retraining
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有的视觉语言模型（VLM）都是针对针孔成像设计的，需要拼接多个窄视场输入才能拼凑出完整的全场景理解。然而，这种多视图感知忽略了单张全景图本身所固有的整体空间与上下文关系。在本工作中，我们提出了全景-语言建模（PLM）范式，这是一种统一的 360° 视觉语言推理，其能力超越其针孔对应物的简单相加。此外，我们提出了 PanoVQA，一个大规模全景视觉问答数据集，涵盖恶劣全景场景，使得在物体遮挡和驾驶事故条件下进行全面的推理成为可能。为给 PLM 奠定基础，我们开发了一个即插即用的全景稀疏注意力模块，使现有的基于针孔的 VLM 无需重新训练即可处理等距柱状全景图。大量实验表明，我们的 PLM 在具有挑战性的全场景下实现了更优的鲁棒性和整体推理能力，其理解能力超越了其窄视场组成部分的简单相加。项目页面：https://github.com/InSAI-Lab/PanoVQA。
英文原文摘要 1103 字符
Existing vision-language models (VLMs) are tailored for pinhole imagery, stitching multiple narrow field-of-view inputs to piece together a complete omni-scene understanding. Yet, such multi-view perception overlooks the holistic spatial and contextual relationships that a single panorama inherently preserves. In this work, we introduce the Panorama-Language Modeling (PLM) paradigm, a unified 360deg vision-language reasoning that is more than the sum of its pinhole counterparts. Besides, we present PanoVQA, a large-scale panoramic VQA dataset that involves adverse omni-scenes, enabling comprehensive reasoning under object occlusions and driving accidents. To establish a foundation for PLM, we develop a plug-and-play panoramic sparse attention module that allows existing pinhole-based VLMs to process equirectangular panoramas without retraining. Extensive experiments demonstrate that our PLM achieves superior robustness and holistic reasoning under challenging omni-scenes, yielding understanding greater than the sum of its narrow parts. Project page: https://github.com/InSAI-Lab/PanoVQA.