SLVMEval：面向文本到长视频生成的合成元评测基准
SLVMEval: Synthetic Meta Evaluation Benchmark for Text-to-Long Video Generation
为文本到长视频生成提供了一套可控的元评测协议：以密集视频描述数据集为源，按 10 个维度合成退化构造「高质量 vs 低质量」成对样本，并用众包过滤出人类可感知的样本对。做视频生成评测或设计自动指标的研究者可直接复用该测试床与成对排序协议来校验自己的评价系统。
原文依据 we synthetically degrade source videos to create controlled "high-quality versus low-quality" pairs across 10 distinct aspects
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文提出合成长视频元评测（SLVMEval），一个用于对文本到视频（T2V）评测系统进行元评测的基准。所提出的 SLVMEval 基准聚焦于在时长最长可达 10,486 秒（约 3 小时）的视频上评估这些系统。该基准针对一项基本要求，即这些系统能否在人类易于评估的设定下准确评估视频质量。我们采用基于成对比较的元评测框架。在密集视频描述数据集的基础上，我们对源视频进行合成退化，围绕 10 个不同维度构造受控的「高质量对低质量」样本对。随后，我们通过众包进行筛选，仅保留退化清晰可感知的样本对，从而建立一个有效的最终测试床。利用该测试床，我们评估现有评测系统对这些样本对进行排序的可靠性。实验结果表明，人类评估者能以 84.7%–96.8% 的准确率识别出更优的长视频，而在 10 个维度中的 9 个上，这些系统的准确率低于人类评估，这揭示了文本到长视频评测中的薄弱之处。
英文原文摘要 1209 字符
This paper proposes the synthetic long-video meta-evaluation (SLVMEval), a benchmark to perform meta-evaluations of text-to-video (T2V) evaluation systems. The proposed SLVMEval benchmark focuses on assessing these systems on videos of up to 10,486 s (approximately 3 h). The benchmark targets a fundamental requirement, i.e., whether the systems can accurately assess video quality in settings that are easy for humans to assess. We adopt a pairwise comparison-based meta-evaluation framework. Building on dense video-captioning datasets, we synthetically degrade source videos to create controlled "high-quality versus low-quality" pairs across 10 distinct aspects. Then, we employ crowdsourcing to filter and retain only those pairs in which the degradation is clearly perceptible, thereby establishing an effective final testbed. Using this testbed, we assess the reliability of existing evaluation systems in ranking these pairs. Experimental results demonstrate that human evaluators can identify the better long video with 84.7%--96.8% accuracy, and in nine of the 10 aspects, the accuracy of these systems falls short of the human assessment, which reveals weaknesses in text-to-long video evaluation.