视觉语言模型 · CVPR2026 ·2026.06.03

DSCA:面向终身VLM编辑的动态子空间概念对齐

DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing

Gyanendra Das、Sai Jena

与你有关的部分

为视觉语言模型提供了一种基于正交语义子空间分解的持续知识编辑方法,可复用于需要顺序更新多模态知识的场景,避免编辑间的相互干扰。

原文依据 We introduce Dynamic Subspace Concept Alignment (DSCA) which by design mitigates this limitation by decomposing the representation space into a set of orthogonal semantic subspaces
可迁移到 视觉语言模型知识编辑持续指令微调多模态推理

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

模型编辑旨在更新知识以添加新概念并修改相关信息,而无需重新训练。终身编辑是一项具有挑战性的任务,容易破坏先前学习的概念,尤其是对于视觉语言模型(VLM),因为顺序编辑可能导致推理能力下降和跨模态对齐失效。现有的基于门控适配器、激活编辑和参数合并技术的VLM知识编辑方法解决了全微调中出现的灾难性遗忘问题;然而,它们仍在VLM的共享表示空间中操作,其中概念相互纠缠,因此编辑会干扰其他不相关的概念。我们假设这种不稳定性持续存在是因为当前方法通过优化算法控制编辑,而非在结构上分离知识。我们引入动态子空间概念对齐(DSCA),该方法通过设计将表示空间分解为一组正交语义子空间,并仅在这些变换后的空间中提出编辑,从而缓解这一限制。这些子空间通过对联合视觉-语言表示进行增量聚类和PCA获得。该过程在结构上隔离概念,通过将隔离从软训练目标转变为架构属性,实现精确、无干扰的编辑。精准编辑由多损失函数指导,以保持任务保真度、编辑局部性和跨模态对齐。在基础模型冻结的情况下,我们的方法实现了98%的单次编辑成功率,在1000次顺序编辑后仍保持超过95%,将幻觉降低3-5%,并在持续指令微调基准上取得了最佳的后向迁移(BWT)分数。大量实验表明,DSCA在各种数据集和基准上的持续终身编辑中具有最优的稳定性和知识保留能力。

英文原文摘要 1893 字符

Model editing aims to update knowledge to add new concepts and change relevant information without retraining. Lifelong editing is a challenging task, prone to disrupting previously learned concepts, especially for Vision Language Models (VLMs), because sequential edits can lead to degraded reasoning and cross-modal misalignment. Existing VLM knowledge editing methods based on gated adapters, activation edits, and parameter merging techniques address catastrophic forgetting seen in full fine-tuning; however, they still operate in the shared representation space of the VLM, where concepts are entangled, so edits interfere with other non-relevant concepts. We hypothesize that this instability persists because current methods algorithmically control edits via optimization rather than structurally separating knowledge. We introduce Dynamic Subspace Concept Alignment (DSCA) which by design mitigates this limitation by decomposing the representation space into a set of orthogonal semantic subspaces and proposing edits only in those transformed spaces. These subspaces are obtained through incremental clustering and PCA on joint vision-language representations. This process structurally isolates concepts, enabling precise, non-interfering edits by turning isolation from a soft training objective into an architectural property. The surgical edits are guided by a multi-term loss function for maintaining task fidelity, edit locality, and cross-modal alignment. With the base model frozen, our method achieves 98% single-edit success, remains over 95% after 1,000 sequential edits, lowers hallucination by 3-5%, and achieves the best backward transfer (BWT) scores on continual instruction-tuning benchmarks. Extensive experiments demonstrate DSCA's state-of-the-art stability and knowledge retention capability in continual lifelong editing across various datasets and benchmarks.

标签

会议 CVPR2026
页码 40772-40781
Bibkey
来源 CVF Open Access