理解视觉语言模型中的任务迁移
Understanding Task Transfer in Vision-Language Models
给做 VLM 感知微调的人一个可复用的诊断工具：用 PGF 归一化指标量化源任务微调对 13 个感知任务零样本性能的迁移广度与幅度，并据此做数据选择，避免负迁移。
原文依据 We introduce Perfection Gap Factor (PGF), a normalized metric that measures change in performance as a result of task transfer.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在多模态基准上表现良好，但在深度估计或目标计数等视觉感知任务上仍落后于人类和专用模型。在一个任务上微调会不可预测地影响其他任务上的性能，这使得针对特定任务的微调颇具挑战。本文通过系统研究任务可迁移性来应对这一挑战。我们考察在一个感知任务上微调 VLM 如何影响其在其他任务上的零样本性能。我们引入完美差距因子（Perfection Gap Factor, PGF），这是一个衡量任务迁移所导致性能变化的归一化指标。我们利用 PGF 计算任务可迁移性，该指标同时刻画了源任务所引发迁移的广度与幅度。我们在 13 个感知任务上评估了三个开放权重 VLM，并构建了任务迁移图，揭示了此前未被观察到的感知任务之间的关系。我们的分析揭示了正迁移与负迁移的模式，识别出相互影响的任务组，依据迁移行为将任务组织为若干人格类型，并展示了 PGF 如何指导数据选择以实现更高效的训练。这些发现既凸显了正迁移的机会，也揭示了负干扰的风险，为推进 VLM 的发展提供了可操作的指导。
英文原文摘要 1351 字符
Vision-Language Models (VLMs) perform well on multimodal benchmarks but lag behind humans and specialized models on visual perception tasks like depth estimation or object counting. Finetuning on one task can unpredictably affect performance on others, making task-specific finetuning challenging. In this paper, we address this challenge through a systematic study of task transferability. We examine how finetuning a VLM on one perception task affects its zero-shot performance on others. We introduce Perfection Gap Factor (PGF), a normalized metric that measures change in performance as a result of task transfer. We utilize PGF to compute Task Transferability, which captures both the breadth and the magnitude of transfer induced by a source task. Using three open-weight VLMs evaluated across 13 perception tasks, we construct a task transfer graph that reveals previously unobserved relationships among perception tasks. Our analysis uncovers patterns of positive and negative transfer, identifies groups of tasks that mutually influence each other, organizes tasks into personas based on their transfer behavior and demonstrates how PGF can guide data selection for more efficient training. These findings highlight both opportunities for positive transfer and risks of negative interference, offering actionable guidance for advancing VLMs.