Spk2VidNet：一种用于从长脉冲相机流中高保真重建视频的分层循环架构
Spk2VidNet: A Hierarchical Recurrent Architecture for High-Fidelity Video Reconstruction from Long Spike-Camera Streams
针对脉冲相机低空间分辨率问题，提出分层循环网络从长脉冲流重建高分辨率图像序列，其分层扩大时序感受野、基于运动一致性的对齐模块和多尺度相似性融合模块可复用于其他时序超分或视频复原任务。
原文依据 we propose a hierarchical recurrent network named Spk2VidNet to reconstruct high-fidelity high resolution image sequences from low resolution spike data
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论脉冲相机是一种具有超高时间分辨率的神经形态视觉传感器，能够通过发射二进制脉冲流来捕捉快速运动场景。然而，其相对较低的空间分辨率限制了精细视觉细节的获取，这推动了脉冲相机超分辨率（SCSR）的研究。现有的SCSR方法通常处理固定长度的脉冲序列，可访问的信息局限于局部时间邻域。此外，脉冲波动阻碍了可靠的强度提取。这两个因素都影响了SCSR的性能。为了解决这些问题，我们提出了一种名为Spk2VidNet的分层循环网络，用于从低分辨率脉冲数据重建高保真高分辨率图像序列。为了减轻波动，Spk2VidNet通过分层扩大时间感受野，逐步利用脉冲流内的时间相关性来增强特征表示。在循环块内，我们引入了一个对齐模块，利用多帧之间的运动一致性来联合估计并相互细化帧间运动，实现更准确的时间对齐。此外，我们开发了一个融合模块，基于多尺度相似性自适应地整合相邻的对齐特征，以实现鲁棒的特征聚合。我们进一步提出了一种带有状态转移策略的分段训练方法，以在有限的GPU内存下高效地建模长期依赖关系。在合成和真实采集的脉冲数据上的实验表明，Spk2VidNet达到了最优性能。
英文原文摘要 1602 字符
Spike camera is a neuromorphic vision sensor with ultra-high temporal resolution, capable of capturing fast-moving scenes by firing a stream of binary spikes. However, its relatively low spatial resolution limits the acquisition of fine-grained visual details, motivating research on spike camera super resolution (SCSR). Existing SCSR methods typically operate on fixed-length spike sequences, where the accessible information is restricted to a local temporal neighborhood. Moreover, spike fluctuations hinder reliable intensity extraction. Both factors affect the performance of SCSR. To address these issues, we propose a hierarchical recurrent network named Spk2VidNet to reconstruct high-fidelity high resolution image sequences from low resolution spike data. To mitigate fluctuations, Spk2VidNet progressively exploits temporal correlations within spike stream to enhance feature representation by hierarchically enlarging temporal receptive fields. Within the recurrent block, we introduce an alignment module that leverages the motion consistency among multiple frames to jointly estimate and mutually refine inter-frame motions, achieving more accurate temporal alignment. In addition, we develop a fusion module to adaptively integrate neighboring aligned features based on multi-scale similarity for robust feature aggregation. We further propose a segment-wise training with state transfer strategy to efficiently model long-term dependencies with limited GPU memory. Experiments on synthetic and real-captured spike data demonstrate that Spk2VidNet achieves state-of-the-art performance.