你在做什么？对可控人体视频生成的深入审视
What Are You Doing? A Closer Look at Controllable Human Video Generation
为可控人体图像到视频生成提供了细粒度基准与改进后的评测指标，做人体视频生成或动作可控生成的研究者可直接用它诊断模型在动作、交互、运动等 9 个维度上的短板，而不必再依赖 TikTok、TED-Talks 这类多样性不足的旧数据。
原文依据 a new benchmark for fine-grained evaluation of controllable image-to-video generation of humans
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论高质量基准是推动机器学习研究进展的关键。然而，尽管视频生成领域关注度日益增长，目前仍缺乏用于评估人体合成的全面数据集。人类能够执行种类繁多的动作与交互，但现有的数据集（如 TikTok、TED-Talks 和 HumanVid）在多样性与复杂性上均不足以充分体现视频生成模型的能力。为填补这一空白，我们提出「你在做什么？」（WYD）：一个用于细粒度评估可控人体图像到视频生成的新基准。WYD 包含 1,544 个带描述文本的视频，这些视频经过精心收集，并以细粒度类别进行标注。这些标注使我们能够系统地衡量模型在人体生成的 9 个方面（包括动作、交互与运动）上的表现。我们还提出了一个评估框架，在其中对现有指标加以改造，以更好地进行人体层面与视频层面的评估，这一点已通过人类偏好得到验证。借助我们的数据集与指标，我们对可控图像到视频生成领域的最优开源模型进行了深入分析，展示了 WYD 如何揭示这些模型能力的新洞见。我们公开数据与代码，以推动人体视频生成领域的进展。
英文原文摘要 1278 字符
High-quality benchmarks are crucial for driving progress in machine learning research. However, despite the growing interest in video generation, there is no comprehensive dataset to evaluate human synthesis. Humans can perform a wide variety of actions and interactions, but existing datasets, like TikTok, TED-Talks, and HumanVid, lack the diversity and complexity to fully capture the capabilities of video generation models. We close this gap by introducing 'What Are You Doing?' (WYD): a new benchmark for fine-grained evaluation of controllable image-to-video generation of humans. WYD consists of 1,544 captioned videos that have been meticulously collected and annotated with fine-grained categories. These allow us to systematically measure performance across 9 aspects of human generation, including actions, interactions and motion. We also propose an evaluation framework, where we adapt existing metrics for better human- and video-level assessment, as shown by human preference. Equipped with our dataset and metrics, we perform in-depth analyses of state-of-the-art open-source models in controllable image-to-video generation, showing how WYD provides novel insights about their capabilities. We release data and code to drive progress in human video generation.