LVLM 辅助的任务特定视觉模型对齐
LVLM-Aided Alignment of Task-Specific Vision Models
用 LVLM 作为「模型行为→自然语言→图像级批评」的双向翻译器，把领域专家给出的类级规范转成对单张图的批评信号来微调小模型，为视觉模型对齐人类知识、去除虚假相关提供了一套无需细粒度标注的可复用流程。
原文依据 provides a bidirectional interface that translates model behavior into natural language and maps human class-level specifications to image-level critiques
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在高风险领域，小型任务特定视觉模型至关重要，因为它们计算需求低，且有大量方法可以解释其结果。然而，这些解释往往揭示出模型与人类领域知识对齐不佳，反而依赖虚假相关。这可能导致模型在真实世界部署后表现出脆弱行为。为解决这一问题，我们提出一种新颖且高效的方法，利用大型视觉语言模型（LVLM）的泛化能力，使小型任务特定视觉模型与人类领域知识对齐。我们的 LVLM 辅助视觉对齐（LVLM-VA）方法提供了一个双向接口，将模型行为翻译为自然语言，并将人类给出的类级规范映射为图像级批评，从而实现领域专家与模型之间的有效交互。在合成数据集和真实世界数据集上的验证表明，我们的方法在使模型行为与人类规范对齐方面取得了显著改进。我们证明，该方法无需细粒度反馈，即可有效降低模型对虚假特征和群体特定偏见的依赖。
英文原文摘要 1207 字符
In high-stakes domains, small task-specific vision models are crucial due to their low computational requirements and the availability of numerous methods to explain their results. However, these explanations often reveal that the models do not align well with human domain knowledge, relying instead on spurious correlations. This might result in brittle behavior once deployed in the real-world. To address this issue, we introduce a novel and efficient method for aligning small task-specific vision models with human domain knowledge by leveraging the generalization capabilities of a Large Vision Language Model (LVLM). Our LVLM-Aided Visual Alignment (LVLM-VA) method provides a bidirectional interface that translates model behavior into natural language and maps human class-level specifications to image-level critiques, enabling effective interaction between domain experts and the model. Our method demonstrates substantial improvement in aligning model behavior with human specifications, as validated on both synthetic and real-world datasets. We show that it effectively reduces the model's dependence on spurious features and on group-specific biases, without requiring fine-grained feedback.