当AVSR遇见视频会议：数据集、退化现象及性能崩溃背后的隐藏机制
When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapse
论文构建了面向视频会议场景的多模态音视频语音识别数据集MLD-VC，并揭示了语音增强算法与Lombard效应引起的分布偏移是性能下降的主因，为视觉研究者提供了真实场景下的鲁棒性评测基准和微调策略。
原文依据 we construct MLD-VC, the first multimodal dataset tailored for VC
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论音视频语音识别（AVSR）在离线条件下已取得显著进展，然而其在真实视频会议（VC）场景中的鲁棒性仍鲜有探索。本文首次对主流VC平台上的最优AVSR模型进行系统评估，揭示了传输失真和自发的人类夸张表情导致的严重性能退化。为弥补这一空白，我们构建了MLD-VC，这是首个面向VC的多模态数据集，包含31位说话人、22.79小时的音视频数据，并明确利用Lombard效应来增强人类夸张表情。通过全面分析，我们发现语音增强算法是分布偏移的主要来源，它会改变音频的第一和第二共振峰。有趣的是，我们发现由Lombard效应引起的分布偏移与语音增强引入的分布偏移高度相似，这解释了为何在Lombard数据上训练的模型在VC中表现出更强的鲁棒性。在MLD-VC上微调AVSR模型可缓解该问题，在多个VC平台上平均降低17.5%的字符错误率（CER）。我们的发现和数据集为开发更鲁棒、更通用的真实视频会议AVSR系统奠定了基础。MLD-VC可在https://huggingface.co/datasets/nccm2p2/MLD-VC获取。
英文原文摘要 1365 字符
Audio-Visual Speech Recognition (AVSR) has achieved remarkable progress in offline conditions, yet its robustness in real-world video conferencing (VC) remains largely unexplored. This paper presents the first systematic evaluation of state-of-the-art AVSR models across mainstream VC platforms, revealing severe performance degradation caused by transmission distortions and spontaneous human hyper-expression. To address this gap, we construct MLD-VC, the first multimodal dataset tailored for VC, comprising 31 speakers, 22.79 hours of audio-visual data, and explicit use of the Lombard effect to enhance human hyper-expression. Through comprehensive analysis, we find that speech enhancement algorithms are the primary source of distribution shift, which alters the first and second formants of audio. Interestingly, we find that the distribution shift induced by the Lombard effect closely resembles that introduced by speech enhancement, which explains why models trained on Lombard data exhibit greater robustness in VC. Fine-tuning AVSR models on MLD-VC mitigates this issue, achieving an average 17.5% reduction in CER across several VC platforms. Our findings and dataset provide a foundation for developing more robust and generalizable AVSR systems in real-world video conferencing. MLD-VC is available at https://huggingface.co/datasets/nccm2p2/MLD-VC.