CineBrain：用于脑条件视频生成的大规模多模态视听脑数据集
CineBrain: A Large-Scale Multi-Modal Audiovisual Brain Dataset for Brain-Conditioned Video Generation
该工作提出的多模态脑信号融合编码器与神经潜变量解码器，为视觉解码提供了可复用的跨模态对齐与融合范式；其同步fMRI-EEG数据集可用于研究听觉皮层激活如何增强视觉解码，对脑机接口和视觉神经科学交叉方向有直接参考价值。
原文依据 we introduce CineBrain, the first large-scale dataset that synchronizes fMRI and EEG during audiovisual viewing
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大多数将脑信号解码为图像的研究（通常将其作为生成模型的先验）仅关注视觉内容。这忽略了大脑整合听觉与视觉信息的自然能力，例如声音会强烈影响我们对视觉场景的感知。为探究这一问题，我们提出了一项新任务：从视听刺激期间记录的多模态脑信号中重建连续视频刺激。为此，我们引入了CineBrain，这是首个在视听观看期间同步采集fMRI和EEG的大规模数据集，包含六小时的《生活大爆炸》剧集用于跨模态对齐。我们还首次系统探索了结合fMRI和EEG进行视频重建，并提出了CineSync框架，该框架使用多模态融合编码器和神经潜变量解码器来重建动态视频。CineSync在动态重建方面达到了最优性能，利用fMRI和EEG的互补优势提升了视觉保真度。我们的分析表明，听觉皮层激活增强了解码准确性，突显了听觉输入在视觉感知中的作用。项目页面：https://jianxgao.github.io/CineBrain。
英文原文摘要 1243 字符
Most research decoding brain signals into images, often using them as priors for generative models, has focused only on visual content. This overlooks the brain's natural ability to integrate auditory and visual information, for instance, sound strongly influences how we perceive visual scenes. To investigate this, we propose a new task of reconstructing continuous video stimuli from multimodal brain signals recorded during audiovisual stimulation. To enable this, we introduce CineBrain, the first large-scale dataset that synchronizes fMRI and EEG during audiovisual viewing, featuring six hours of The Big Bang Theory episodes for cross-modal alignment. We also conduct the first systematic exploration of combining fMRI and EEG for video reconstruction and present CineSync, a framework for reconstructing dynamic video using a Multi-Modal Fusion Encoder and a Neural Latent Decoder. CineSync achieves state-of-the-art performance in dynamic reconstruction, leveraging the complementary strengths of fMRI and EEG to improve visual fidelity. Our analysis shows that auditory cortical activations enhance decoding accuracy, highlighting the role of auditory input in visual perception. Project Page: https://jianxgao.github.io/CineBrain.