Frame2Freq：面向细粒度视频理解的频谱适配器
Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding
把图像预训练骨干迁移到视频时，用沿时间轴的 FFT 做频带特定嵌入，替代单尺度时域适配器，可复用于任何需要多时间尺度运动建模的视频下游任务（如细粒度动作识别、时序定位）。
原文依据 Frame2Freq uses Fast Fourier Transform (FFT) along time and learns frequency-band specific embeddings
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论将图像预训练骨干网络适配到视频，通常依赖针对单一时间尺度调优的时域适配器。我们的实验表明，这些模块捕捉的是静态图像线索和极快的闪烁变化，却忽略了中速运动。然而，跨多个时间尺度捕捉动态对于细粒度时序分析（例如打开与关闭瓶子）至关重要。为解决这一问题，我们提出 Frame2Freq——一族频率感知适配器，在预训练视觉基础模型（VFM）的图像到视频适配过程中进行频谱编码，从而提升细粒度动作识别。Frame2Freq 沿时间轴使用快速傅里叶变换（FFT），并学习频带特定的嵌入，以自适应地突出最具判别性的频率范围。在五个细粒度活动识别数据集上，Frame2Freq 优于先前的 PEFT 方法，并在其中四个数据集上甚至超过了全量微调模型。这些结果为频率分析方法是在图像到视频迁移中建模时序动态的有力工具提供了令人鼓舞的证据。
英文原文摘要 1111 字符
Adapting image-pretrained backbones to video typically relies on time-domain adapters tuned to a single temporal scale. Our experiments show that these modules pick up static image cues and very fast flicker changes, while overlooking medium-speed motion. Capturing dynamics across multiple time-scales is, however, crucial for fine-grained temporal analysis (i.e., opening vs. closing bottle).To address this, we introduce Frame2Freq -- a family of frequency-aware adapters that perform spectral encoding during image-to-video adaptation of pretrained Vision Foundation Models (VFMs), improving fine-grained action recognition. Frame2Freq uses Fast Fourier Transform (FFT) along time and learns frequency-band specific embeddings that adaptively highlight the most discriminative frequency ranges. Across five fine-grained activity recognition datasets, Frame2Freq outperforms prior PEFT methods and even surpasses fully fine-tuned models on four of them. These results provide encouraging evidence, that frequency analysis methods are a powerful tool for modeling temporal dynamics in image-to-video transfer.