爱屋及乌：重新思考标签在视觉上下文学习提示词检索中的作用
Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning
为视觉上下文学习中的提示词检索引入标签一致性约束，并设计图像-标签联合表示与查询自适应混合专家路由，可直接用于分割、检测等下游任务的提示词选择。
原文依据 We reveal these approaches sometimes get visually similar but label-inconsistent prompts, which potentially degrade VICL performance.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉上下文学习（VICL）通过示范性提示词引导视觉基础模型处理多种任务。此类提示词的选择在很大程度上影响VICL性能，因而成为关键挑战。先前工作在提示词检索与重排序策略上取得了实质性进展，但主要关注提示图像而忽略了标签。我们揭示这些方法有时会得到视觉相似但标签不一致的提示词，这可能降低VICL性能。另一方面，查询与提示词之间更高的标签一致性更可能表明更强的VICL结果。受这些发现启发，我们开发了一个名为LaPR（标签感知提示词检索）的框架，强调标签在提示词选择中的作用。我们的框架首先为提示词设计图像-标签联合表示，以显式融入标签线索。此外，为处理测试时查询标签不可用的情况，我们向双编码器引入混合专家机制，并采用查询自适应路由。每个专家预期捕获特定的标签模式，而路由器推断查询自适应的混合权重并帮助学习标签感知表示。我们精心设计了专家与路由器的交替优化，分别采用VICL性能引导的对比损失和标签引导的对比损失。大量实验表明，LaPR在上下文分割、检测和着色任务上取得了显著且一致的改进。此外，LaPR在特征提取器和交叉折叠场景中均表现出良好的泛化能力，表明标签利用在VICL提示词检索中的重要性。代码见https://github.com/luotc-why/CVPR26-LaPR。
英文原文摘要 1734 字符
Visual in-context learning (VICL) enables visual foundation models to handle multiple tasks by steering them with demonstrative prompts. The choice of such prompts largely influences VICL performance, standing out as a key challenge. Prior work has made substantial progress on prompt retrieval and reranking strategies, but mainly focuses on prompt images while overlooking labels. We reveal these approaches sometimes get visually similar but label-inconsistent prompts, which potentially degrade VICL performance. On the other hand, higher label consistency between query and prompts preferably indicates stronger VICL results. Motivated by these findings, we develop a framework named LaPR (Label-aware Prompt Retrieval), which highlights the role of labels in prompt selection. Our framework first designs an image-label joint representation for prompts to incorporate label cues explicitly. Besides, to handle unavailable query labels at test time, we introduce a mixture-of-expert mechanism to the dual encoders with query-adaptive routing. Each expert is expected to capture a specific label mode, while the router infers query-adaptive mixture weights and helps to learn label-aware representation. We carefully design alternative optimization for experts and router, with a VICL performance-guided contrastive loss and a label-guided contrastive loss, respectively. Extensive experiments show promising and consistent improvement of LaPR on in-context segmentation, detection, and colorization tasks. Moreover, LaPR generalizes well across feature extractors and cross-fold scenarios, suggesting the importance of label utilization in prompt retrieval for VICL. Code is available at https://github.com/luotc-why/CVPR26-LaPR.