StructXLIP：利用多模态结构线索增强视觉-语言模型
StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues
StructXLIP 将边缘图作为视觉结构代理，并设计结构中心损失来增强视觉-语言对齐，为跨模态检索提供了一种即插即用的微调策略，可复用于其他多模态模型的结构感知训练。
原文依据 We introduce StructXLIP, a fine-tuning alignment paradigm that extracts edge maps (e.g., Canny), treating them as proxies for the visual structure of an image
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于边缘的表征是视觉理解的基本线索，这一原理根植于早期视觉研究，至今仍处于核心地位。我们将这一原理扩展到视觉-语言对齐中，表明在模态间分离并对齐结构线索可以极大地有利于对长而细节丰富的描述进行微调，并特别关注改进跨模态检索。我们引入 StructXLIP，一种微调对齐范式，它提取边缘图（例如 Canny），将其视为图像视觉结构的代理，并过滤相应的描述以强调结构线索，使其成为“以结构为中心”的描述。微调在标准对齐损失的基础上增加了三个以结构为中心的损失：(i) 将边缘图与结构文本对齐，(ii) 将局部边缘区域与文本块匹配，(iii) 将边缘图与彩色图像连接以防止表征漂移。从理论角度来看，标准 CLIP 最大化视觉和文本嵌入之间的互信息，而 StructXLIP 额外最大化多模态结构表征之间的互信息。这种辅助优化本质上更难，引导模型走向更稳健和语义稳定的极小值，从而增强视觉-语言对齐。除了在通用和专用领域的跨模态检索上优于当前竞争对手之外，我们的方法还可作为一种通用的提升方案，能够以即插即用的方式集成到未来的方法中。代码和预训练模型可在 https://github.com/intelligolabs/StructXLIP 获取。
英文原文摘要 1616 字符
Edge-based representations are fundamental cues for visual understanding, a principle rooted in early vision research and still central today. We extend this principle to vision-language alignment, showing that isolating and aligning structural cues across modalities can greatly benefit fine-tuning on long, detail-rich captions, with a specific focus on improving cross modal retrieval. We introduce StructXLIP, a fine-tuning alignment paradigm that extracts edge maps (e.g., Canny), treating them as proxies for the visual structure of an image, and filters the corresponding captions to emphasize structural cues, making them "structure-centric". Fine-tuning augments the standard alignment loss with three structure-centric losses: (i) aligning edge maps with structural text, (ii) matching local edge regions to textual chunks, and (iii) connecting edge maps to color images to prevent representation drift. From a theoretical standpoint, while standard CLIP maximizes the mutual information between visual and textual embeddings, StructXLIP additionally maximizes the mutual information between multimodal structural representations. This auxiliary optimization is intrinsically harder, guiding the model toward more robust and semantically stable minima, enhancing vision-language alignment. Beyond outperforming current competitors on cross-modal retrieval on both general and specialized domains, our method serves as a general boosting recipe that can be integrated into future approaches in a plug-and-play manner. Code and pretrained models are available at: https://github.com/intelligolabs/StructXLIP.