基于手语数据的文本驱动三维手部动作生成
Text-Driven 3D Hand Motion Generation from Sign Language Data
把手语视频的伪标注通过 LLM 转成细粒度手部动作文本描述，从而以极大规模自动构建「3D 手部动作-文本」配对，并用文本条件扩散模型生成手部运动；对做人体/手部动作生成、手语理解与文本到运动的研究者，这套数据构造与跨域评测设定可直接复用。
原文依据 we leverage a large-scale sign language video dataset, along with noisy pseudo-annotated sign categories
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们的目标是训练一个三维手部动作的生成模型，以自然语言描述为条件，这些描述指明手型、位置、手指/手/手臂运动等动作特征。为此，我们以空前的规模自动构建了三维手部动作与其对应文本标签的配对数据。具体而言，我们利用一个大规模手语视频数据集以及带噪声的伪标注手语类别，借助一个大语言模型（该模型使用手语属性词典以及我们补充的动作-文本线索）将这些类别转译为手部动作描述。该数据使得训练一个文本条件的手部动作扩散模型（HandMDM）成为可能，该模型在跨域情形下具有鲁棒性，例如同一手语中未见过的类别、另一种手语中的手语，以及非手语的手部动作。我们针对这些情形开展了大量实验研究，并将公开我们训练好的模型与数据，以支持这一相对新兴领域的未来研究。
英文原文摘要 1029 字符
Our goal is to train a generative model of 3D hand motions, conditioned on natural language descriptions specifying motion characteristics such as handshapes, locations, finger/hand/arm movements. To this end, we automatically build pairs of 3D hand motions and their associated textual labels with unprecedented scale. Specifically, we leverage a large-scale sign language video dataset, along with noisy pseudo-annotated sign categories, which we translate into hand motion descriptions via an LLM that utilizes a dictionary of sign attributes, as well as our complementary motion-script cues. This data enables training a text-conditioned hand motion diffusion model (HandMDM), that is robust across domains such as unseen sign categories from the same sign language, but also signs from another sign language and non-sign hand movements. We contribute extensive experimental investigation of these scenarios and will make our trained models and data publicly available to support future research in this relatively new field.