mVLM：面向 mNPU 的视觉语言模型
mVLM: A Vision Language Model for mNPUs
面向 NPU 端侧部署的轻量 VLM 架构：用纯 NPU 友好算子构建动态卷积编码器（全局空间调制）与 SSM 解码器，并配合编码器-解码器全参数协同量化，为视觉研究者提供在极端内存约束下压缩 VLM 并保持生成质量的工程与架构思路。
原文依据 we introduce a coordinated full-parameter quantization strategy that preserves precision across the encoder-decoder boundary
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论集成神经处理单元（NPU）的低功耗智能处理器（称为 mNPU）的普及，为端侧生成式人工智能创造了新机遇，使智能可穿戴设备和小型机器人等终端设备受益。然而，严格的内存约束和有限的算子支持严重阻碍了视觉语言模型（VLM）在 mNPU 上的部署。为弥合这一关键差距，我们提出 mVLM，这是首个面向 mNPU 设计的轻量化 VLM 架构。它由我们提出的 OverMod 编码器和 AttSSM 解码器组成。OverMod 是一个受仿生视觉启发的轻量动态卷积网络，融合了我们新颖的全局空间调制机制，仅使用 NPU 友好的算子即可实现自适应、高保真的特征提取。AttSSM 利用高效的状态空间模型（SSM）核心，并辅以多尺度特征融合和全局上下文动态调制机制，以执行稳健的序列建模。此外，我们引入了一种协同全参数量化策略，在编码器-解码器边界上保持精度，并针对 SSM 等不受支持的模块手工优化了算子。mVLM 在 COCO Karpathy 测试划分上取得了具有竞争力的 117.8 CIDEr 分数，并首次展示了在 mNPU 平台上实现毫秒级 VLM 推理的可行性。
英文原文摘要 1389 字符
The proliferation of low-power intelligent processors with integrated Neural Processing Units (NPUs), called mNPUs, has created new opportunities for on-device generative AI, benefitting end devices like smart wearables and small robots. However, deploying Vision-Language Models (VLMs) on mNPUs is severely hindered by stringent memory constraints and limited operator support. To bridge this critical gap, we propose mVLM, the first lightweight-oriented VLM architecture designed for mNPUs. It is comprised of our proposed OverMod encoder and AttSSM decoder. OverMod is a lightweight dynamic convolutional network inspired by biomimetic vision, incorporating our novel Global Spatial Modulation mechanism to enable adaptive, high-fidelity feature extraction using only NPU-friendly operators. AttSSM leverages a highly efficient State Space Model (SSM) core, augmented with multi-scale feature fusion and Global Context Dynamic Modulation mechanism, to perform robust sequential modeling. Furthermore, we introduce a coordinated full-parameter quantization strategy that preserves precision across the encoder-decoder boundary, alongside hand-optimized operators for unsupported modules like SSMs. mVLM achieves a competitive CIDEr score of 117.8 on the COCO Karpathy test split and, for the first time, demonstrates the feasibility of millisecond-level VLM inference on a mNPU platform.