OpenFS：具备隐式手语手检测与帧级字母条件合成的多手手指拼写识别
OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis
提出隐式手语手检测与单调对齐损失，可复用于手语识别、手势理解等需要时序对齐与手部定位的任务；帧级字母条件生成器为数据增强与合成基准构建提供思路。
原文依据 The SF loss encourages cross-attention to focus on the signing hand, enabling implicit signing-hand detection during recognition.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论手指拼写是手语的一个组成部分，其中单词通过特定的手部姿势逐字母拼出。自动手指拼写识别在弥合聋人与听人社区之间的沟通鸿沟方面起着至关重要的作用，然而由于手语手歧义问题、缺乏合适的训练损失以及词汇外（OOV）问题，它仍然具有挑战性。先前的手指拼写识别方法依赖于显式手语手检测，这常常导致识别失败，并且依赖于连接主义时序分类（CTC）损失，该损失表现出峰值行为问题。为了解决这些问题，我们开发了OpenFS，一种用于手指拼写识别与合成的开源方法。我们提出了一种支持单手和多手输入的多手手指拼写识别器，并通过结合双层级位置编码和手语手聚焦（SF）损失来执行隐式手语手检测。SF损失鼓励交叉注意力聚焦于手语手，从而在识别过程中实现隐式手语手检测。此外，在不依赖CTC损失的情况下，我们引入了一种单调对齐（MA）损失，该损失通过交叉注意力正则化强制输出字母序列遵循输入姿势序列的时序顺序。另外，我们提出了一种帧级字母条件生成器，用于为OOV单词合成逼真的手指拼写姿势序列。该生成器能够构建一个新的合成基准，称为FSNeo。通过全面的实验，我们证明了我们的方法在识别方面达到了最优性能，并验证了所提出的识别器和生成器的有效性。代码和数据可在以下网址获取：https://github.com/AIRC-KETI/OpenFS。
英文原文摘要 1825 字符
Fingerspelling is a component of sign languages in which words are spelled out letter by letter using specific hand poses. Automatic fingerspelling recognition plays a crucial role in bridging the communication gap between Deaf and hearing communities, yet it remains challenging due to the signing-hand ambiguity issue, the lack of appropriate training losses, and the out-of-vocabulary (OOV) problem. Prior fingerspelling recognition methods rely on explicit signing-hand detection, which often leads to recognition failures, and on a connectionist temporal classification (CTC) loss, which exhibits the peaky behavior problem. To address these issues, we develop OpenFS, an open-source approach for fingerspelling recognition and synthesis. We propose a multi-hand-capable fingerspelling recognizer that supports both single- and multi-hand inputs and performs implicit signing-hand detection by incorporating a dual-level positional encoding and a signing-hand focus (SF) loss. The SF loss encourages cross-attention to focus on the signing hand, enabling implicit signing-hand detection during recognition. Furthermore, without relying on the CTC loss, we introduce a monotonic alignment (MA) loss that enforces the output letter sequence to follow the temporal order of the input pose sequence through cross-attention regularization. In addition, we propose a frame-wise letter-conditioned generator that synthesizes realistic fingerspelling pose sequences for OOV words. This generator enables the construction of a new synthetic benchmark, called FSNeo. Through comprehensive experiments, we demonstrate that our approach achieves state-of-the-art performance in recognition and validate the effectiveness of the proposed recognizer and generator. Codes and data are available in: https://github.com/AIRC-KETI/OpenFS.