SemVideo：通过分层语义引导从大脑活动中重建你所观看的内容
SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance
把分层语义提示（静态锚框描述、运动叙述、整体摘要）作为条件注入视频生成，并用三分支注意力融合重建运动模式，这套「语义引导 + 运动解耦」的思路可迁移到视频预测、条件视频生成与跨模态对齐任务。
原文依据 a Motion Adaptation Decoder that reconstructs dynamic motion patterns using a novel tripartite attention fusion architecture
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从大脑活动中重建动态视觉体验，为探索人类视觉感知的神经机制提供了一条极具吸引力的途径。尽管基于 fMRI 的图像重建近期取得了显著进展，但将这一成功扩展到视频重建仍是一项重大挑战。当前的 fMRI 到视频重建方法始终面临两大缺陷：(i) 显著对象在帧间的视觉表征不一致，导致外观失配；(ii) 时序连贯性差，造成运动错位或帧间突变。为解决这些局限，我们提出 SemVideo，一个由分层语义信息引导的新型 fMRI 到视频重建框架。SemVideo 的核心是 SemMiner，这是一个分层引导模块，从原始视频刺激中构建三个层级的语义线索：静态锚框描述、面向运动的叙述以及整体摘要。借助这一语义引导，SemVideo 包含三个关键组件：语义对齐解码器，将 fMRI 信号与由 SemMiner 导出的 CLIP 风格嵌入进行对齐；运动适配解码器，利用一种新颖的三分支注意力融合架构重建动态运动模式；以及条件视频渲染器，利用分层语义引导进行视频重建。在 CC2017 与 HCP 数据集上开展的实验表明，SemVideo 在语义对齐与时序一致性两方面均取得更优性能，刷新了 fMRI 到视频重建的最优结果。
英文原文摘要 1611 字符
Reconstructing dynamic visual experiences from brain activity provides a compelling avenue for exploring the neural mechanisms of human visual perception. While recent progress in fMRI-based image reconstruction has been notable, extending this success to video reconstruction remains a significant challenge. Current fMRI-to-video reconstruction approaches consistently encounter two major shortcomings: (i) inconsistent visual representations of salient objects across frames, leading to appearance mismatches; (ii) poor temporal coherence, resulting in motion misalignment or abrupt frame transitions.To address these limitations, we introduce SemVideo, a novel fMRI-to-video reconstruction framework guided by hierarchical semantic information. At the core of SemVideo is SemMiner, a hierarchical guidance module that constructs three levels of semantic cues from the original video stimulus: static anchor descriptions, motion-oriented narratives, and holistic summaries. Leveraging this semantic guidance, SemVideo comprises three key components: a Semantic Alignment Decoder that aligns fMRI signals with CLIP-style embeddings derived from SemMiner, a Motion Adaptation Decoder that reconstructs dynamic motion patterns using a novel tripartite attention fusion architecture, and a Conditional Video Render that leverages hierarchical semantic guidance for video reconstruction. Experiments conducted on the CC2017 and HCP datasets demonstrate that SemVideo achieves superior performance in both semantic alignment and temporal consistency, setting a new state-of-the-art in fMRI-to-video reconstruction.