无需文本学习有效手语特征的无注释手语翻译
Learning Effective Sign Features without Text for Gloss-free Sign Language Translation
SignDINO 将 DINO 的自蒸馏框架与局部掩码视图结合，仅用全局帧推理即可获得对细微运动敏感的表征，这一「全局-局部掩码对齐」策略可迁移到其他需要细粒度运动建模的视觉任务（如动作识别、微表情分析）的预训练中。
原文依据 a student-teacher architecture is employed, where the teacher model receives the global sign frame, while the student model learns from masked local views
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论自监督学习（SSL）在自然语言处理与计算机视觉领域均取得了显著成功。然而，手语翻译（SLT）模型在预训练阶段仍严重依赖基于注释的 SLT 中的注释标注或无注释 SLT（GFSLT）中的文本标注，以确保骨干网络为翻译模型提供有效的手语（SL）特征。这种依赖限制了 SLT 模型的可扩展性与泛化能力。一个自然的问题随之产生：能否将现有的 SSL 方法直接应用于手语领域，训练出有效的特征提取器用于下游 GFSLT 任务，从而消除对文本标注的需求？本文提出一个简单而有效的预训练框架，其目标有二：（1）将预训练过程与注释或文本标注解耦，仅依赖手语帧；（2）推理阶段仅需全局帧以简化流程。我们表明，直接应用现有 SSL 方法会得到次优性能，因为手语特征涉及细微的运动模式与判别性线索，这些往往局限于局部区域。为此，我们引入 SignDINO，一种简单而有效的手语感知 DINO 训练策略，能够在无任何文本监督的情况下从全局帧中学习有效且语义有意义的表征。具体而言，采用学生-教师架构，教师模型接收全局手语帧，而学生模型从仅保留手部与面部区域的掩码局部视图中学习。这种简单设计促使模型从判别性局部线索推断全局语义，使得教师模型在推理时仅基于全局视图即可提取与手语相关的特征。在公开手语数据集上的大量实验表明，SignDINO 在不依赖额外线索或额外手语相关预训练的情况下，在 GFSLT 任务上取得了极具竞争力的性能。
英文原文摘要 1950 字符
Self-supervised learning (SSL) has achieved remarkable success across both NLP and CV domains. However, sign language translation (SLT) models still heavily rely on gloss annotations in gloss-based SLT or text annotations in gloss-free SLT (GFSLT) during pretraining, aiming to ensure that the backbone provides effective sign language (SL) features for the translation model. Such reliance restricts the scalability and generalization ability of the SLT model. One natural question arises: Can existing SSL methods be directly applied to the SL domain to train an effective sign feature extractor for downstream GFSLT tasks, eliminating the need for text annotations? In this paper, we propose a simple yet effective pretraining framework with two goals: (1) decoupling the pretraining process from gloss or text annotations, relying purely on sign frames; and (2) only global frames are required during inference for simplicity. We show that directly applying existing SSL methods yields suboptimal performance, as SL features involve subtle motion patterns and discriminative cues that are often confined to local regions. To achieve this, we introduce SignDINO, a simple yet effective sign-aware DINO training strategy that learns effective and semantically meaningful representations from global frames without any textual supervision. Specifically, a student-teacher architecture is employed, where the teacher model receives the global sign frame, while the student model learns from masked local views that preserve only the hand and facial regions. Such a simple design encourages the model to infer global semantics from discriminative local cues, allowing the teacher model to extract SL-related feature during inference solely based on global views. Extensive experiments on public SL datasets show that SignDINO achieves highly competitive performance on the GFSLT task without relying on extra cues or additional SL-related pretraining.