通过多标签原型增长和双线索一致性的测试时自我-他人中心适应用于动作预测
Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency
提出测试时自适应框架，利用多标签原型增长和双线索一致性，在无需目标域数据训练的情况下实现跨视角动作预测，可复用于其他存在域偏移的视频理解任务。
原文依据 we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE^ 2 A^ 3 ) task
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论高效适应自我中心（Ego）和他人中心（Exo）视角对于人机协作等应用至关重要。然而，大多数现有的Ego-Exo适应方法的成功严重依赖于目标视角数据进行训练，从而增加了计算和数据收集成本。本文首次探索了用于动作预测的测试时Ego-Exo适应（TE^2A^3）任务，旨在测试时在线调整源视角训练的模型以预测目标视角动作。由于多动作候选和显著的时空跨视角差距，现有测试时自适应（TTA）方法难以解决该任务。因此，我们提出了一种新颖的双线索增强原型增长网络（DCPGN），它积累多标签知识并整合跨模态线索，以实现有效的测试时Ego-Exo适应和动作预测。具体而言，我们提出了多标签原型增长模块（ML-PGM），通过多标签分配和基于置信度的重加权来平衡多个正类，用于类级记忆库，这些记忆库通过熵优先队列策略更新。然后，双线索一致性模块（DCCM）引入了一个轻量级叙述器来生成指示动作进展的文本线索，以补充包含各种对象的视觉线索。此外，我们约束推断的文本和视觉logits以构建双线索一致性，从而在时间和空间上桥接Ego和Exo视角。在新提出的EgoMe-anti和现有EgoExoLearn基准上的大量实验表明了我们的方法的有效性，其大幅优于相关的最优方法。代码可在https://github.com/ZhaofengSHI/DCPGN获取。
英文原文摘要 1851 字符
Efficient adaptation between Egocentric (Ego) and Exocentric (Exo) views is crucial for applications such as human-robot cooperation. However, the success of most existing Ego-Exo adaptation methods relies heavily on target-view data for training, thereby increasing computational and data collection costs. In this paper, we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE^ 2 A^ 3 ) task, which aims to adjust the source-view-trained model online during test time to anticipate target-view actions. It is challenging for existing Test-Time Adaptation (TTA) methods to address this task due to the multi-action candidates and significant temporal-spatial inter-view gap. Hence, we propose a novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation. Specifically, we propose a Multi-Label Prototype Growing Module (ML-PGM) to balance multiple positive classes via multi-label assignment and confidence-based reweighting for class-wise memory banks, which are updated by an entropy priority queue strategy. Then, the Dual-Clue Consistency Module (DCCM) introduces a lightweight narrator to generate textual clues indicating action progressions, which complement the visual clues containing various objects. Moreover, we constrain the inferred textual and visual logits to construct dual-clue consistency for temporally and spatially bridging Ego and Exo views. Extensive experiments on the newly proposed EgoMe-anti and the existing EgoExoLearn benchmarks show the effectiveness of our method, which outperforms related state-of-the-art methods by a large margin. Code is available at \href https://github.com/ZhaofengSHI/DCPGN https://github.com/ZhaofengSHI/DCPGN .