解耦视觉与语言：码本锚定的视觉自适应
Decoupling Vision and Language: Codebook Anchored Visual Adaptation
用离散码本把视觉编码器输出锚定到稳定词元空间，使编码器可独立微调而不改动投影器或 LLM，为视觉侧域自适应提供了一种与语言架构解耦的可复用方案。
原文依据 fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（LVLM）利用其视觉编码器将图像转换为表示，以供下游推理使用，但这些编码器在医学图像诊断或细粒度分类等特定领域视觉任务上往往表现不佳，此时表示误差会通过语言模型逐级传播，导致回答错误。现有的自适应方法通过投影器调优或其他参数高效更新来修改编码器与语言模型之间的连续特征接口，这仍然将两个组件耦合在一起，并且每当编码器发生变化时都需要重新对齐。我们提出 CRAFT（Codebook RegulAted Fine-Tuning），一种轻量级方法，它使用离散码本对编码器进行微调，将视觉表示锚定到稳定的词元空间，从而在不修改模型其他部分的情况下实现域自适应。这种解耦设计使得经过自适应的编码器能够无缝提升具有不同语言架构的 LVLM 的性能，只要它们共享同一码本。在实验中，CRAFT 在 VQARAD 和 PlantVillage 等 10 个特定领域基准上平均提升 13.51%，同时保持 LLM 的语言能力，并优于在连续词元上操作的同类方法。
英文原文摘要 1298 字符
Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens.