面向模态缺失 RGBT 跟踪的时空条件去噪 Transformer
Spatio-Temporal Conditional Denoising Transformer for Modality-Missing RGBT Tracking
把扩散式条件去噪引入多模态跟踪的特征重建：用长短期时序上下文作为条件，把可用模态的噪声特征逐步去噪成时序一致的多模态表征，并用噪声调制适配机制在单一网络内统一缺失/完整模态两种输入，可直接迁移到其他模态缺失的多模态融合任务。
原文依据 SCDT leverages the short-term temporal cues from recent historical frames to capture the fine-grained temporal correlations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论RGBT 跟踪中的模态缺失往往导致多模态特征表征不完整且不稳定，从而严重降低性能。现有方法通常尝试从可用模态中恢复缺失模态，但在挑战性场景下生成数据的质量可能并不理想。此外，当前方法在处理缺失数据与完整数据时灵活性有限。为克服这些局限，我们提出时空条件去噪 Transformer（SCDT），它在统一框架中融合空间线索与时序上下文，自适应地对缺失模态进行信息重建、对弱模态进行特征增强，从而实现鲁棒的模态缺失 RGBT 跟踪。具体而言，SCDT 利用来自近期历史帧的短期时序线索来捕捉细粒度的时序相关性，并利用编码模态演化的长期时序线索来捕捉全局上下文。通过联合利用长短期时序上下文作为条件，SCDT 逐步引导可用模态的含噪特征学习可靠且时序一致的多模态表征。此外，SCDT 引入噪声调制适配机制，根据模态可用情况动态调整其行为，使得单一框架无需改变结构或参数即可统一模态缺失与完整场景下的特征学习。在三个公开基准数据集上的大量实验表明，我们的方法持续优于最优方法。
英文原文摘要 1639 字符
Missing modalities in RGBT tracking often lead to incomplete and unstable multimodal feature representations that greatly degrade the performance. Existing methods typically attempt to recover missing modalities from available ones, but the quality of data generated in challenging scenarios might be unsatisfactory. In addition, current approaches exhibit limited flexibility in processing both missing and complete data. To overcome these limitations, we propose a Spatio-temporal Conditional Denoising Transformer (SCDT), which integrates the spatial cues and the temporal context to adaptively perform information reconstruction of missing modalities and feature enhancement of weak modalities in a unified framework, for robust modality-missing RGBT tracking. In particular, SCDT leverages the short-term temporal cues from recent historical frames to capture the fine-grained temporal correlations and the long-term temporal cues encoding modality evolution to capture the global context. By jointly exploiting long short-term temporal contexts as the conditions, SCDT progressively guides noisy features of available modalities to learn reliable and temporally consistent multimodal representations. Furthermore, SCDT introduces a noise-modulated adaptation mechanism that dynamically adjusts its behavior according to the modal availability, enabling a single framework to unify feature learning under both modality-missing and complete scenarios without changing the architecture or parameters. Extensive experiments on three public benchmark datasets demonstrate that our method consistently outperforms state-of-the-art methods.