文本引导特征解耦的跨模态步态识别
Text-guided Feature Disentanglement for Cross-modal Gait Recognition
用 LLM 生成的模态感知文本先验作为语义锚点,配合 CLIP 多粒度编码器把激光雷达点云序列与 RGB 视频对齐到统一视觉-语言空间,再通过 top-k 文本匹配与残差正交约束解耦出模态共享步态特征;这套「文本引导解耦」思路可迁移到其他 2D-3D 跨模态检索与识别任务。
原文依据 leverages modality-aware textual priors as semantic anchors to guide the learning of disentangled modality-shared representations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论步态识别是一种基于行走模式识别个体身份的生物特征技术,在远距离、非侵入式场景中具有优势。然而,现实场景常涉及激光雷达与 RGB 相机等异构传感模态,由于 2D 视频与 3D 点云序列之间存在巨大的模态鸿沟,激光雷达-相机跨模态步态识别(LCCGR)成为一项关键而富有挑战的任务。为应对该挑战,我们提出 TCFDNet,即文本引导的跨模态特征解耦网络,它利用模态感知的文本先验作为语义锚点,引导解耦后的模态共享表征的学习。具体而言,我们利用大语言模型构建步态模态文本字典(GMTD),生成跨模态与跨视角的丰富步态语义描述。基于 CLIP 的多粒度特征编码器随后在统一的视觉-语言空间中对齐视觉与文本特征。此外,文本引导特征解耦(TFD)模块选取匹配度最高的前 k 个文本描述来重建模态特定表征,并通过残差分解与正交约束导出模态共享特征。为缓解解耦共享特征的脆弱性,我们提出特征稳定性增强(FSE)模块,通过建模空间与通道相关性提升特征鲁棒性。此外,引入跨模态图像块交换策略以进一步提升泛化能力。在 SUSTech1K 与 FreeGait 数据集上的大量实验表明,TCFDNet 取得了新的最优结果,并验证了所提各模块的有效性。
英文原文摘要 1725 字符
Gait recognition is a biometric technique that identifies individuals based on their walking patterns, offering advantages in long-range, non-intrusive scenarios. However, real-world scenarios often involve heterogeneous sensing modalities such as LiDAR and RGB cameras, making LiDAR-Camera Cross-modal Gait recognition (LCCGR) a critical yet challenging task due to the substantial modality gap between 2D videos and 3D point cloud sequences. To address this challenge, we propose TCFDNet, a Text-guided Cross-modal Feature Disentanglement Network, which leverages modality-aware textual priors as semantic anchors to guide the learning of disentangled modality-shared representations. Specifically, we construct a Gait Modality Text Dictionary (GMTD) using large language models to generate rich semantic descriptions of gait across modalities and viewpoints. A CLIP-based Multi-grained Feature Encoder then aligns visual and textual features within a unified vision-language space. Furthermore, the Text-guided Feature Disentanglement (TFD) module selects the top\text - k matched textual descriptions to reconstruct modality-specific representations and derive modality-shared features via residual decomposition and orthogonality constraints. To mitigate the fragility of the disentangled shared features, we propose a Feature Stability Enhancement (FSE) module, which models spatial and channel-wise correlations to improve feature robustness. In addition, a cross-modal patch exchange strategy is introduced to further improve generalization. Extensive experiments on SUSTech1K and FreeGait datasets demonstrate that TCFDNet achieves new state-of-the-art results and validate the effectiveness of the proposed modules.