连接大脑与语义：面向语义增强 fMRI 视频重建的分层框架
Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstruction
把 fMRI 信号映射到同时包含文本语义、图像内容、动作概念与物体类别的联合嵌入空间，再用记忆库检索融合来约束视频生成，这套「语义富化 + 记忆检索」的两阶段思路可迁移到脑机解码之外的弱条件视频生成与跨模态检索增强生成。
原文依据 a bottom-up semantic enrichment stage maps fMRI signals to a rich embedding space that comprehensively captures textual semantics, image contents, action concepts, and object categories
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从功能性磁共振成像（fMRI）重建动态视觉体验为视频，对推进神经过程的理解至关重要。然而，当前的 fMRI 到视频重建方法受制于噪声 fMRI 信号与丰富视频内容之间的语义鸿沟，其根源在于依赖不完整的语义嵌入——这些嵌入既无法捕捉视频特有的线索（如动作），也未整合先验知识。为此，我们受人类大脑双通路加工机制的启发，提出 CineNeuron，一个通过两个协同阶段从 fMRI 信号进行语义增强视频重建的新型分层框架。首先，自底向上的语义富化阶段将 fMRI 信号映射到一个丰富的嵌入空间，该空间全面捕捉文本语义、图像内容、动作概念与物体类别。其次，自顶向下的记忆整合阶段利用所提出的记忆混合方法，从先前见过的数据中动态选取相关「记忆」，并将其与 fMRI 嵌入融合以精化视频重建。在两个 fMRI 到视频基准上的大量实验结果表明，CineNeuron 在多项指标上超越了最优方法。
英文原文摘要 1277 字符
Reconstructing dynamic visual experiences as videos from functional magnetic resonance imaging (fMRI) is pivotal for advancing the understanding of neural processes. However, current fMRI-to-video reconstruction methods are hindered by a semantic gap between noisy fMRI signals and the rich content of videos, stemming from a reliance on incomplete semantic embeddings that neither capture video-specific cues (e.g., actions) nor integrate prior knowledge. To this end, we draw inspiration from the dual-pathway processing mechanism in human brain and introduce CineNeuron, a novel hierarchical framework for semantically enhanced video reconstruction from fMRI signals with two synergistic stages. First, a bottom-up semantic enrichment stage maps fMRI signals to a rich embedding space that comprehensively captures textual semantics, image contents, action concepts, and object categories. Second, a top-down memory integration stage utilizes the proposed Mixture-of-Memories method to dynamically select relevant "memories" from previously seen data and fuse them with the fMRI embedding to refine the video reconstruction. Extensive experimental results on two fMRI-to-video benchmarks demonstrate that CineNeuron surpasses state-of-the-art methods across various metrics.