在多模态大语言模型中用事件流学习看穿极端光照
Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models
把事件流作为与 RGB 互补的模态接入 MLLM，并用 DINOv2 分支导出的可学习光照指示信号自适应调制融合权重，同时用潜空间对齐损失把退化特征拉回正常光照语义——这套「退化感知的模态门控 + 潜空间语义对齐」可直接迁移到低光、过曝、夜间等视觉任务的特征融合设计中。
原文依据 an Illumination Indicator -- a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLM）在标准条件下具备强大的视觉-语言推理能力，但在极端光照下会失效，此时 RGB 输入会不可逆地丢失结构与语义。我们提出 Event-MLLM，一个事件增强模型，通过动态融合事件流与 RGB 帧来实现全光照条件下的视觉推理。两个关键组件驱动了我们的方法：光照指示器——一个由 DINOv2 分支导出的可学习信号，用于表征曝光退化并自适应地调制事件-RGB 融合；以及光照校正损失，它在潜变量空间中将融合特征与非退化（正常光照）语义对齐，以补偿极端光照下丢失的信息。我们构建了首个面向 MLLM 的多光照事件指令语料库，包含 2,241 个事件-RGB 样本（每个样本约 6 组问答对），覆盖多样场景与 17 档亮度倍率（0.05 倍至 20 倍），并提供了一个用于极端光照下推理、计数与细粒度识别的指令跟随基准。实验表明，Event-MLLM 显著优于通用、光照自适应以及仅用事件的基线方法，在挑战性光照下的鲁棒多模态感知与推理方面树立了新的技术水平。
英文原文摘要 1255 字符
Multimodal Large Language Models (MLLMs) perform strong vision-language reasoning under standard conditions but fail in extreme illumination, where RGB inputs lose irrevocable structure and semantics. We propose Event-MLLM, an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames. Two key components drive our approach: an Illumination Indicator -- a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion -- and an Illumination Correction Loss that aligns fused features with non-degraded (normal-light) semantics in the latent space, compensating for information lost in extreme lighting. We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples (around 6 QA pairs each) across diverse scenes and 17 brightness rates (0.05x - 20x), plus an instruct-following benchmark for reasoning, counting, and fine-grained recognition under extreme lighting. Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines, setting a new state of the art in robust multimodal perception and reasoning under challenging illumination.