面向大型视觉语言模型的语言引导频率调制
Language-guided Frequency Modulation for Large Vision-Language Models
在视觉 token 送入 LLM 之前，用语言引导在频域上对视觉特征做高通/低通分解与选择性增强，为视觉侧提供了一种无需额外训练参数、可插拔的细粒度表征调控手段，可直接迁移到任意 LVLM 的视觉编码器-投影器链路上。
原文依据 a plug-and-play approach that adaptively refines visual signals in the frequency domain under linguistic guidance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（LVLM）在多种任务的视觉推理中展现出卓越能力。这些任务对视觉表征提出了不同要求：有些侧重高层次的全局上下文，有些则强调细粒度的局部细节。然而，现有大多数方法主要在空间域上处理视觉表征，缺乏显式机制来区分高频局部细节与低频全局上下文。这一局限阻碍了对视觉表征的细粒度控制，也使其与语言的分层对齐变得复杂。为解决该问题，我们提出语言引导频率调制（LFM），一种即插即用的方法，在语言引导下于频域中自适应地精炼视觉信号。通过有选择地增强关键区域与细节，LFM 实现了更具结构性与精确性的视觉处理。关键在于，它不引入额外的训练参数，仅依赖一个轻量级可学习投影器在视觉 token 融入 LLM 之前对其进行精炼，从而确保极小的计算开销。在多种视觉-语言基准上的大量实验凸显了 LFM 的可扩展性、有效性以及对 LVLM 的广泛适用性。
英文原文摘要 1284 字符
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in visual reasoning across diverse tasks. These tasks place different demands on visual representations: some prioritize high-level global context, while others emphasize fine-grained local details. However, most existing methods operate on visual representations primarily in the spatial domain, lacking an explicit mechanism for distinguishing between high-frequency local details and low-frequency global context. This limitation hinders fine-grained control of visual representations and complicates their hierarchical alignment with language. To address this issue, we introduce Language-guided Frequency Modulation (LFM), a plug-and-play approach that adaptively refines visual signals in the frequency domain under linguistic guidance. By selectively enhancing critical regions and details, LFM enables more structured and precise visual processing. Crucially, it adds no extra training parameters, relying solely on a lightweight learnable projector to refine visual tokens before integration into the LLM, thereby ensuring minimal computational overhead. Extensive experiments across diverse vision-language benchmarks highlight LFM's scalability, effectiveness, and broad applicability to LVLMs.