统一多模态大模型的罗塞塔石碑：一种破译理解与生成的统一分词器
Rosetta Stone For Unified MLLMs: A Unified Tokenizer to Decipher Understanding and Generation
提出一种分层解耦代理任务并引入注意力优先映射的统一视觉分词器，可直接替换离散统一多模态大模型中的视觉分词模块，在理解与生成任务上同时提升；其关于对比学习式模型在统一生成与理解时存在内在冲突、以及码本收敛行为差异的分析，对设计统一视觉表征具有直接参考价值。
原文依据 we hierarchically decouple the conflicting proxy tasks, enriching the diversity of semantic features supervision
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当前主流的统一分词器大多以像素重建和特征对齐作为预训练任务，但在架构、监督目标与任务交互等关键维度上仍基本未被探索，可能导致性能受限。我们系统性地考察了统一视觉分词器的关键因素，并提出一种新框架，从多个方面增强理解与生成之间的协同。我们最初的分析聚焦于前沿视觉模型的特性，确认了对比学习式模型在统一生成与理解时存在内在冲突，并揭示了码本具有不同的收敛行为。为解决上述瓶颈，我们对相互冲突的代理任务进行分层解耦，丰富语义特征监督的多样性，以增强语义能力与低层能力。随后，我们进一步引入注意力优先映射策略，以强大的语义先验引导细粒度生成。在 256×256 分辨率下，我们的方法在 ImageNet 上取得 0.33 的 rFID 和 80.9% 的零样本准确率，超过 VILA-U 7.6%，并优于 SigLIP 的连续嵌入。将其应用于离散统一多模态大模型时，我们的 7B 模型在理解任务上超过 TokenFlow-13B 3.1%，并在 GenAI-Bench 与 MJHQ-30K 上取得最优性能。
英文原文摘要 1378 字符
Major state-of-the-art unified tokenizers predominantly adopt pixel reconstruction and feature alignment as pretext tasks, they leave key domains largely unexplored such as architecture, supervised objectives and tasks interaction, potentially resulting in limited performance. We systematically investigate the critical factors of a unified visual tokenizer and propose a novel framework that strengthens synergy between understanding and generation in various aspects. Our initial analysis focus on properties of frontier vision models, confirming inherent conflict in contrastive learning style models for unifying generation and understanding, and demonstrate distinct convergence behavior of codebooks. To address the above bottleneck, we hierarchically decouple the conflicting proxy tasks, enriching the diversity of semantic features supervision to enhance thesemantic and low-level capabilities. Subsequently, we further introduce attention-prioritized mapping strategy, which guides fine-grained generation with powerful semantic prior. Our method achieves rFID of 0.33 and zero-shot accuracy of 80.9% on ImageNet at 256x256 resolution, surpassing VILA-U by 7.6% and outperforms continuous embedding of SigLIP. When applied to discrete unified MLLMs, our 7B model exceeds TokenFlow-13B by 3.1% in understanding and achieve SOTA performance in GenAI-Bench and MJHQ-30K.