SAVA-X:基于场景自适应视角对齐与双向跨视角融合的第一人称到第三人称模仿误差检测
SAVA-X: Ego-to-Exo Imitation Error Detection via Scene-Adaptive View Alignment and Bidirectional Cross View Fusion
面向第一人称与第三人称异步、长度不匹配视频的跨视角时序对齐与融合框架,其视图条件自适应采样和双向交叉注意力可直接迁移到 egocentric-exocentric 视频配对、跨视角动作定位与步骤级质量评估等任务。
原文依据 given asynchronous, length-mismatched ego and exo videos, the model must localize procedural steps on the ego timeline
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论误差检测在工业培训、医疗保健和装配质量控制中至关重要。现有工作大多假设单视角设定,无法处理用第三人称(exo)演示来评估第一人称(ego)模仿这一实际情形。我们形式化了第一人称到第三人称(Ego->Exo)模仿误差检测任务:给定异步且长度不匹配的第一人称与第三人称视频,模型需在第一人称时间轴上定位各个流程步骤,并判定每一步是否存在错误。该设定带来了跨视角域偏移、时序错位以及严重冗余等挑战。在统一协议下,我们适配了来自密集视频描述与时序动作检测的强基线方法,并表明它们在这一跨视角情形下表现不佳。随后我们提出 SAVA-X,一个“对齐—融合—检测”框架,包含(i)视图条件自适应采样、(ii)场景自适应视角嵌入,以及(iii)双向交叉注意力融合。在 EgoMe 基准上,SAVA-X 在 AUPRC 与平均 tIoU 上稳定优于所有基线,消融实验证实了其各组成部分的互补收益。
英文原文摘要 1080 字符
Error detection is crucial in industrial training, healthcare, and assembly quality control. Most existing work assumes a single-view setting and cannot handle the practical case where a third-person (exo) demonstration is used to assess a first-person (ego) imitation. We formalize Ego->Exo Imitation Error Detection: given asynchronous, length-mismatched ego and exo videos, the model must localize procedural steps on the ego timeline and decide whether each is erroneous. This setting introduces cross-view domain shift, temporal misalignment, and heavy redundancy. Under a unified protocol, we adapt strong baselines from dense video captioning and temporal action detection and show that they struggle in this cross-view regime. We then propose SAVA-X, an Align-Fuse-Detect framework with (i) view-conditioned adaptive sampling, (ii) scene-adaptive view embeddings, and (iii) bidirectional cross-attention fusion. On the EgoMe benchmark, SAVA-X consistently improves AUPRC and mean tIoU over all baselines, and ablations confirm the complementary benefits of its components.