HiFICL：面向多模态任务的高保真上下文学习
HiFICL: High-Fidelity In-Context Learning for Multimodal Tasks
把 ICL 中示例的影响精确分解为注意力输出与上下文值的动态混合，并用可学习的虚拟键值对加低秩分解来替代「shift vector」近似，这给视觉-语言模型提供了一种免示例、低开销的任务适配手段，可直接嵌入 LMM 的注意力层做端到端训练。
原文依据 a set of "virtual key-value pairs" to act as a learnable context
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论上下文学习（ICL）是大型多模态模型（LMM）的一种重要范式，它利用少量上下文示例（ICD）来适配新任务。然而，其性能对示例配置较为敏感，且计算开销高昂。从数学上看，这些示例的影响可以分解为标准注意力输出与上下文值的一种动态混合。现有的近似方法通过学习一个「偏移向量」来简化这一过程。受该精确分解的启发，我们提出高保真上下文学习（HiFICL），以更忠实的方式对 ICL 机制进行建模。HiFICL 由三个关键部分组成：1）一组「虚拟键值对」，充当可学习的上下文；2）一种低秩分解，用于稳定且带正则化的训练；3）一个简单的端到端训练目标。从另一个角度看，该机制构成了一种上下文感知的参数高效微调（PEFT）。大量实验表明，HiFICL 在多个多模态基准上持续优于现有的近似方法。
英文原文摘要 1097 字符
In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HiFICL) to more faithfully model the ICL mechanism. HiFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks.