VQRAE：面向多模态理解、生成与重建的表征量化自编码器
VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
提出用高维语义VQ码本（1536维）在统一分词器中同时输出连续语义特征和离散token，视觉研究者可借鉴其两阶段训练策略（先冻结编码器学码本，再自蒸馏联合优化）来构建兼顾理解与生成的视觉表征。
原文依据 we propose VQRAE, a Vector Quantization version of Representation AutoEncoders, which pioneers the first exploration in unified representation to produce Continuous semantic features for image understanding and Discrete tokens for visual generation within a unified tokenizer.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在单一分词器中统一多模态理解、生成与重建表征，仍是构建统一模型的关键挑战。先前研究主要尝试以双编码器范式解决该问题，例如分别使用独立编码器处理理解与生成，或通过对比损失平衡语义表征与低层特征。本文提出VQRAE，即表征自编码器的向量量化版本，首次探索在统一分词器内同时产生用于图像理解的连续语义特征和用于视觉生成的离散token的统一表征。具体而言，我们基于预训练视觉基础模型构建对称ViT解码器，并采用两阶段训练策略：首先冻结编码器，以像素重建为目标学习高维语义VQ码本；然后通过自蒸馏约束联合优化编码器。该设计使得语义信息几乎无损以保持多模态理解能力，离散token兼容生成，并实现细粒度重建。此外，我们发现了量化语义编码器时依赖高维码本的有趣特性，这与先前图像重建中常用低维码本的做法相反。语义VQ码本在1536维时可达100%利用率。VQRAE在视觉理解、生成与重建的多个基准上展现出有竞争力的性能，并因其离散优势在自回归范式中具有可扩展性。
英文原文摘要 1717 字符
Unifying multimodal understanding, generation and reconstruction representation in a single tokenizer remains a key challenge in building unified models. Previous research predominantly attempts to address this in a dual encoder paradigm, e.g., utilizing the separate encoders for understanding and generation respectively or balancing semantic representations and low-level features with contrastive loss. In this paper, we propose VQRAE, a Vector Quantization version of Representation AutoEncoders, which pioneers the first exploration in unified representation to produce Continuous semantic features for image understanding and Discrete tokens for visual generation within a unified tokenizer. Specifically, we build upon pretrained vision foundation models with a symmetric ViT decoder and adopt a two-stage training strategy: first, it freezes the encoder and learns a high-dimensional semantic VQ codebook with pixel reconstruction objective; then jointly optimizes the encoder with self-distillation constraints. This design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens that are compatible for generation and fine-grained reconstruction. Besides, we identify the intriguing property in quantizing semantic encoders that rely on high-dimensional codebook in contrast to the previous common practice of low-dimensional codebook in image reconstruction. The semantic VQ codebook can achieve a 100% utilization ratio at a dimension of 1536. VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction with promising scaling property in the autoregressive paradigm for its discrete merits.