RE-VLM：事件增强的视觉语言模型用于场景理解
RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding
将事件相机作为补充模态引入视觉语言模型，通过双流编码器和渐进式训练对齐异构特征，为低光照、高动态范围等挑战性场景下的视觉语言理解提供了可复用的多模态融合方案。
原文依据 We propose RE-VLM, the first dual-stream vision-language model that jointly leverages RGB images and event streams for robust scene understanding
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论传统视觉语言模型（VLM）难以解读在恶劣条件下（如低光照、高动态范围或快速运动）拍摄的场景，因为标准RGB图像在这些环境中会退化。事件相机提供了一种互补模态：它们以高时序分辨率和宽动态范围异步记录每个像素的亮度变化，在帧图像失效时保留运动线索。我们提出RE-VLM，这是首个双流视觉语言模型，联合利用RGB图像和事件流，在正常和挑战性条件下实现鲁棒的场景理解。RE-VLM采用并行的RGB和事件编码器，以及渐进式训练策略，将异构视觉特征与语言对齐。为解决RGB-事件-文本监督稀缺的问题，我们进一步提出一种图驱动的流程，将同步的RGB-事件流转换为可验证的场景图，并从中合成描述和问答（QA）对。为开发和评估RE-VLM，我们构建了两个数据集：针对光照挑战场景的PEOD-Chat和覆盖多样场景的RGBE-Chat。在描述和VQA基准上，RE-VLM在参数量相当的情况下持续优于最优的纯RGB和纯事件模型，在挑战性条件下增益尤为显著。这些结果证明了事件增强VLM在广泛真实环境中实现鲁棒视觉语言理解的有效性。代码和数据集见https://github.com/bupt-ai-cz/RE-VLM。
英文原文摘要 1601 字符
Conventional vision-language models (VLMs) struggle to interpret scenes captured under adverse conditions (e.g., low light, high dynamic range, or fast motion) because standard RGB images degrade in such environments. Event cameras provide a complementary modality: they asynchronously record per-pixel brightness changes with high temporal resolution and wide dynamic range, preserving motion cues where frames fail. We propose RE-VLM, the first dual-stream vision-language model that jointly leverages RGB images and event streams for robust scene understanding across both normal and challenging conditions. RE-VLM employs parallel RGB and event encoders together with a progressive training strategy that aligns heterogeneous visual features with language. To address the scarcity of RGB-Event-Text supervision, we further propose a graph-driven pipeline that converts synchronized RGB-Event streams into verifiable scene graphs, from which we synthesize captions and question-answer (QA) pairs. To develop and evaluate RE-VLM, we construct two datasets: PEOD-Chat, targeting illumination-challenged scenes, and RGBE-Chat, covering diverse scenarios. On captioning and VQA benchmarks, RE-VLM consistently outperforms state-of-the-art RGB-only and event-only models with comparable parameter counts, with particularly large gains under challenging conditions. These results demonstrate the effectiveness of event-augmented VLMs in achieving robust vision-language understanding across a wide range of real-world environments. Code and datasets are available at https://github.com/bupt-ai-cz/RE-VLM.