BioVITA：面向视觉-文本-声学对齐的生物数据集、模型与基准
BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment
把音频作为第三模态接入 BioCLIP2 式的图文对比表征空间，并给出覆盖三模态全方向、三个分类层级的跨模态检索基准，为视觉研究者提供可复用的多模态对齐训练范式与评测协议。
原文依据 we introduce a two-stage training framework to effectively align audio representations with visual and textual representations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从多模态数据中理解动物物种，是计算机视觉与生态学交叉领域的一项新兴挑战。尽管近期的生物模型（如 BioCLIP）已在物种识别的图像与文本分类学信息之间展现出强对齐能力，但音频模态的整合仍是一个未解问题。我们提出 BioVITA，一个面向生物应用的新型视觉-文本-声学对齐框架。BioVITA 包含三部分：(i) 一个训练数据集，(ii) 一个表征模型，(iii) 一个检索基准。首先，我们构建了一个大规模训练数据集，包含 130 万个音频片段与 230 万张图像，覆盖 14,133 个物种，并标注了 34 个生态性状标签。其次，我们在 BioCLIP2 的基础上引入两阶段训练框架，以有效地将音频表征与视觉、文本表征对齐。第三，我们构建了一个跨模态检索基准，覆盖三种模态之间所有可能的定向检索（即图像到音频、音频到文本、文本到图像及其反向方向），并涵盖科、属、种三个分类层级。大量实验表明，我们的模型学习到了一个统一的表征空间，能够捕捉超越分类学的物种级语义，推动了多模态生物多样性理解。项目主页见：https://dahlian00.github.io/BioVITA_Page/
英文原文摘要 1428 字符
Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology.While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding. The project page is available at: https://dahlian00.github.io/BioVITA_Page/