DeAR：通过分解注意力头角色实现细粒度 VLM 适配
DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles
把 VLM 适配的粒度从「层」下沉到「注意力头」，用 Concept Entropy 给每个头打上属性/泛化/混合的角色标签，再用 Role-Based Attention Mask 隔离任务知识与泛化能力——做提示词学习或参数高效微调时，可直接复用这套头级角色划分与掩码控制信息流的思路，缓解适配与零样本泛化的权衡。
原文依据 we introduce a novel metric, Concept Entropy, to systematically classify attention heads into distinct functional roles
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论提示词学习是将预训练视觉语言模型（VLM）适配到下游任务的主流范式。然而，现有方法往往依赖一种以层为中心的简化视角，假设浅层捕获通用特征、深层处理任务特定知识。这一假设导致可学习 token 与原始 token 之间产生不受控的交互。任务特定知识可能损害模型的核心泛化能力，并在任务适配与零样本泛化保持之间形成权衡。为解决这一问题，我们挑战以层为中心的视角，提出 DeAR，一个通过分解注意力头角色（Decomposing Attention head Roles）实现细粒度 VLM 适配的框架。我们认为，VLM 内部的功能专门化并不发生在层与层之间，而是发生在更深层中单个注意力头这一更细粒度层面。基于这一洞见，我们引入一种新度量——概念熵（Concept Entropy），将注意力头系统地划分为不同的功能角色：属性（Attribute）、泛化（Generalization）与混合（Mixed）。在这些角色的引导下，我们引入专门的属性 token 以及基于角色的注意力掩码机制来精确控制信息流，确保泛化头与任务特定知识保持隔离。我们进一步引入任务自适应融合策略用于推理。在十五个数据集上的大量实验表明，DeAR 在任务适配与泛化之间取得了良好的平衡，在多种任务上优于先前方法。
英文原文摘要 1558 字符
Prompt learning is a dominant paradigm for adapting pre-trained Vision-Language Models (VLMs) to downstream tasks. However, existing methods often rely on a simplistic, layer-centric view, assuming shallow layers capture general features while deep layers handle task-specific knowledge. This assumption results in uncontrolled interactions between learnable tokens and original tokens. Task-specific knowledge could degrade the model's core generalization and creates a trade-off between task adaptation and the preservation of zero-shot generalization. To address this, we challenge the layer-centric view and propose **DeAR**, a framework that achieves fine-grained VLM adaptation by **De**composing **A**ttention head **R**oles. We posit that the functional specialization within VLMs occurs not between layers, but at the finer-grained level of individual attention heads in the deeper layers. Based on this insight, we introduce a novel metric, Concept Entropy, to systematically classify attention heads into distinct functional roles: *Attribute*, *Generalization*, and *Mixed*. Guided by these roles, we introduce specialized attribute tokens and a Role-Based Attention Mask mechanism to precisely control information flow, ensuring generalization heads remain isolated from task-specific knowledge. We further incorporate a Task-Adaptive Fusion Strategy for inference. Extensive experiments on fifteen datasets show that DeAR achieves a strong balance between task adaptation and generalization, outperforming previous methods across various tasks.