视觉-语音模型：教语音模型谈论图像
Vision-Speech Models: Teaching Speech Models to Converse about Images
为预训练语音模型通过轻量适配模块注入视觉理解能力，并设计动态门控机制在视觉输入与无关对话间切换，为构建实时多模态语音对话系统提供可复用方案。
原文依据 augmenting a recent dialogue speech LLM, Moshi, with visual inputs through lightweight adaptation modules
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉-语言模型近期的成功引出了一个问题：如何等效地让预训练语音模型具备视觉理解能力，这是构建能够自由谈论图像的多模态语音模型的重要里程碑。构建这样的对话式视觉-语音模型带来了独特挑战：(i) 配对的图像-语音数据集比图像-文本数据集稀缺得多；(ii) 确保推理时的实时延迟至关重要，因此带来计算和内存约束；(iii) 模型应保留无法仅从文本推断的韵律特征（例如说话者语调）。在这项工作中，我们提出了 MoshiVis，通过轻量适配模块为近期的对话语音大语言模型 Moshi 增加视觉输入。额外的动态门控机制使模型能够更轻松地在视觉输入和无关对话主题之间切换。为了降低训练成本，我们设计了一个简单的一阶段、参数高效的微调流程，其中我们利用图像-文本（即“无语音”）和图像-语音样本的混合。我们在下游视觉理解任务上使用音频和文本提示评估模型，并报告与 MoshiVis 交互的定性样本。
英文原文摘要 1269 字符
The recent successes of Vision-Language models raise the question of how to equivalently imbue a pretrained speech model with vision understanding, an important milestone towards building a multimodal speech model able to freely converse about images. Building such a conversational Vision-Speech model brings its unique challenges: (i) paired image-speech datasets are much scarcer than their image-text counterparts, (ii) ensuring real-time latency at inference is crucial thus bringing compute and memory constraints, and (iii) the model should preserve prosodic features (e.g., speaker tone) which cannot be inferred from text alone. In this work, we introduce MoshiVis, augmenting a recent dialogue speech LLM, Moshi, with visual inputs through lightweight adaptation modules. An additional dynamic gating mechanism enables the model to more easily switch between the visual inputs and unrelated conversation topics. To reduce training costs, we design a simple one-stage, parameter-efficient fine-tuning pipeline in which we leverage a mixture of image-text (i.e., "speechless") and image-speech samples. We evaluate the model on downstream visual understanding tasks with both audio and text prompts, and report qualitative samples of interactions with MoshiVis.