在基础模型时代重新审视模型拼接
Revisiting Model Stitching In the Foundation Model Era
提出在目标模型倒数第二层使用简单特征匹配损失训练拼接层，使异构视觉基础模型（如 CLIP、DINOv2、SigLIP2）可靠拼接，为多模态大模型共享早期层、灵活组合不同 VFM 优势提供可复用技术。
原文依据 With a simple feature-matching loss at the target model's penultimate layer, heterogeneous VFMs become reliably stitchable across vision tasks.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论模型拼接通过一个轻量拼接层将一个模型（源模型）的早期层与另一个模型（目标模型）的后期层连接起来，已被用作探测表征兼容性的手段。先前工作发现，在同一数据集上训练的模型尽管初始化或目标不同，仍然可以拼接（精度下降可忽略）。我们重新审视视觉基础模型（VFM）的拼接问题，这些模型在目标、数据和模态上各不相同（例如 CLIP、DINOv2、SigLIP2），并追问：异构 VFM 是否可拼接？我们引入了一个系统性的协议，涵盖拼接位置、拼接层族、训练损失和下游任务。由此得出三个发现。（1）拼接层的训练至关重要：在拼接位置匹配中间特征或端到端优化任务损失的传统方法难以保持精度，尤其是在较浅的拼接位置。（2）在目标模型的倒数第二层使用简单的特征匹配损失，异构 VFM 便可在各种视觉任务上可靠地拼接。（3）对于较深的拼接位置，拼接后的模型可以显著超越任一组成模型，且推理开销很小（仅拼接层）。基于这些发现，我们进一步提出 VFM 拼接树（VST），它在多个 VFM 之间共享早期层而保留各自的后期层，为经常利用多个 VFM 的多模态大语言模型提供了可控的精度-延迟权衡。综上所述，我们的研究将拼接从一种诊断性探测提升为一种实用方案，用于整合互补的 VFM 优势并 pinpoint 它们的表征在何处对齐或分歧。
英文原文摘要 1635 字符
Model stitching, connecting early layers of one model (source) to later layers of another (target) via a light stitch layer, has served as a probe of representational compatibility. Prior work finds that models trained on the same dataset remain stitchable (negligible accuracy drop) despite different initializations or objectives. We revisit stitching for Vision Foundation Models (VFMs) that vary in objectives, data, and modality (e.g., CLIP, DINOv2, SigLIP2) and ask: Are heterogeneous VFMs stitchable? We introduce a systematic protocol spanning stitch positions, stitch layer families, training losses, and downstream tasks. Three findings emerge. (1) Stitch layer training matters: conventional approaches that match the intermediate features at the stitch position or optimize the task loss end-to-end struggle to retain accuracy, especially at shallow stitch positions. (2) With a simple feature-matching loss at the target model's penultimate layer, heterogeneous VFMs become reliably stitchable across vision tasks. (3) For deep stitch positions, the stitched model can significantly surpass either constituent model with a small inference overhead (for the stitch layer). Building on these findings, we further propose the VFM Stitch Tree (VST), which shares early layers across VFMs while retaining their later layers, yielding a controllable accuracy-latency trade-off for multimodal LLMs that often leverage multiple VFMs. Taken together, our study elevates stitching from a diagnostic probe to a practical recipe for integrating complementary VFM strengths and pinpointing where their representations align or diverge.