连接异构协同感知中的模态隔离
Linking Modality Isolation in Heterogeneous Collaborative Perception
针对多智能体协同感知中「模态隔离」（不同模态智能体从未在同一帧共现）这一设定，用码本把各模态特征空间正则化为离散码空间，再学习跨模态 feature-code-feature 翻译实现无需空间重叠监督的对齐；做 V2X/多车协同感知、跨模态特征对齐或模态缺失鲁棒性的人可直接复用该「无共现对齐」思路与极低通信量的码传输方案。
原文依据 we propose CodeAlign, the first efficient, co-occurrence-free alignment framework that smoothly aligns modalities via cross-modal feature-code-feature(FCF) translation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论协同感知通过多个智能体之间的数据交换来提升整体感知能力。然而，智能体之间的异构性会引入阻碍协作的域间差距，而一个尚未被充分研究的问题进一步加剧了这一差距：模态隔离。当具有不同模态的多个智能体在任何训练数据帧中都从未共同出现时，就会出现模态隔离，从而扩大跨模态域间差距。现有的对齐方法依赖空间重叠观测提供的监督，因此无法处理模态隔离。为应对这一挑战，我们提出 CodeAlign，这是首个高效、无需共现的对齐框架，通过跨模态的特征—码—特征（FCF）翻译来平滑地对齐模态。其核心思想是通过码本显式地识别表征一致性，并直接学习模态特定特征空间之间的映射，从而无需空间对应关系。码本将特征空间正则化为码空间，提供紧凑而富有表现力的表征。在为每个模态准备好码空间后，CodeAlign 学习 FCF 翻译，将特征映射到其他模态对应的码，再解码回目标码空间中的特征，从而实现有效对齐。实验表明，在整合三种模态时，CodeAlign 仅需先前对齐方法 8% 的训练参数，将通信负载降低 1024 倍，并在 OPV2V 和 DAIR-V2X 数据集上取得最优的感知性能。代码将公开。
英文原文摘要 1591 字符
Collaborative perception leverages data exchange among multiple agents to enhance overall perception capabilities. However, heterogeneity across agents introduces domain gaps that hinder collaboration, and this is further exacerbated by an underexplored issue: modality isolation. It arises when multiple agents with different modalities never co-occur in any training data frame, enlarging cross-modal domain gaps. Existing alignment methods rely on supervision from spatially overlapping observations, thus fail to handle modality isolation. To address this challenge, we propose CodeAlign, the first efficient, co-occurrence-free alignment framework that smoothly aligns modalities via cross-modal feature-code-feature(FCF) translation. The key idea is to explicitly identify the representation consistency through codebook, and directly learn mappings between modality-specific feature spaces, thereby eliminating the need for spatial correspondence. Codebooks regularize feature spaces into code spaces, providing compact yet expressive representations. With a prepared code space for each modality, CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space, enabling effective alignment. Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior alignment methods, reduces communication load by 1024x, and achieves state-of-the-art perception performance on both OPV2V and DAIR-V2X dataset. Code will be released.