通过因果发现与双模态安全子空间投影诊断并修复视觉语言模型中的不安全通道
Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection
提供了一套可复用的诊断-修复范式:先用因果中介分析定位导致不安全行为的神经元与层,再通过广义特征分解在良性与恶意激活之间学习视觉/文本双模态安全子空间,并在推理时动态投影激活。做多模态安全对齐或激活引导的研究者可直接借用该子空间构造与混合融合机制来抑制不安全特征而不损伤语义保真度。
原文依据 we introduce a dual-modal safety subspace projection method that learns generalized safety subspaces for both visual and textual modalities
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型(LVLM)在多模态理解与推理任务上取得了令人瞩目的性能,然而其内部安全机制仍然不透明且难以控制。在本工作中,我们提出了一个用于诊断并修复 LVLM 内部不安全通道的综合框架。我们首先进行因果中介分析,以识别对不安全行为具有因果责任的神经元与层。基于这些发现,我们引入了一种双模态安全子空间投影方法,通过在良性激活与恶意激活之间进行广义特征分解,为视觉与文本两种模态学习泛化的安全子空间。在推理过程中,激活通过一种混合融合机制被动态地投影至这些安全子空间,该机制自适应地平衡视觉与文本修正,从而在保持语义保真度的同时有效抑制不安全特征。在多个 LVLM 安全基准上的大量实验表明,我们的因果-子空间修复框架在不降低通用多模态能力的前提下显著增强了安全鲁棒性,优于先前的激活引导与基于对齐的基线方法。此外,我们的方法展现出稳健的迁移能力,能够有效防御未见过的攻击与自适应攻击。
英文原文摘要 1324 字符
Large Vision-Language Models (LVLMs) have achieved impressive performance across multimodal understanding and reasoning tasks, yet their internal safety mechanisms remain opaque and poorly controlled. In this work, we present a comprehensive framework for diagnosing and repairing unsafe channels within LVLMs. We first perform causal mediation analysis to identify neurons and layers that are causally responsible for unsafe behaviors. Based on these findings, we introduce a dual-modal safety subspace projection method that learns generalized safety subspaces for both visual and textual modalities through generalized eigen-decomposition between benign and malicious activations. During inference, activations are dynamically projected toward these safety subspaces via a hybrid fusion mechanism that adaptively balances visual and textual corrections, effectively suppressing unsafe features while preserving semantic fidelity.Extensive experiments on multiple LVLM safety benchmarks demonstrate that our causal-subspace repair framework significantly enhances safety robustness without degrading general multimodal capabilities, outperforming prior activation steering and alignment-based baselines. Moreover, our approach demonstrates robust transferability, effectively defending against unseen and adaptive attacks.