LLaMo：以连续自回归 token 扩展预训练语言模型实现统一的动作理解与生成
LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens
把人体动作编码进因果连续潜空间、用轻量 flow-matching 头在 decoder-only 骨干上保留下一 token 预测，从而绕开离散量化的抖动并实现 30FPS 流式生成——这套「连续 token + MoT 隔离模态」的做法可直接迁移到视频/动作序列的视觉-语言统一建模。
原文依据 We encode human motion into a causal continuous latent space and maintain the next-token prediction paradigm in the decoder-only backbone through a lightweight flow-matching head
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大模型的最新进展推动了统一多模态生成与理解的显著进步。然而，将动作-语言生成与理解统一起来的模型发展仍基本处于空白。现有方法往往在成对的动作文本数据上微调大语言模型（LLM），由于可用的文本-动作配对数据规模有限，这会导致语言能力的灾难性遗忘。此外，先前方法通常通过量化把动作转换为离散表示以与语言模型集成，离散 token 化会引入明显的抖动伪影。为应对这些挑战，我们提出 LLaMo，一个通过模态特定的 Transformer 混合（MoT）架构扩展预训练 LLM 的统一框架。该设计从根本上保留了基模型的语言理解能力，同时实现可扩展的多模态适配。我们将人体动作编码进因果连续潜空间，并通过轻量的流匹配头在仅解码器骨干网络中保持下一 token 预测范式，从而实现实时的流式动作生成（>30 FPS）。借助预训练 LLM 全面的语言理解能力与大规模动作-文本预训练，实验表明 LLaMo 在通用设定下实现了高保真的文本到动作生成与动作到文本描述，尤其是在零样本动作生成方面，标志着向通用统一动作-语言大模型迈出的重要一步。项目主页：https://kunkun0w0.github.io/project/LLaMo/
英文原文摘要 1646 字符
Recent progress in large models has led to significant advances in unified multimodal generation and understanding. However, the development of models that unify motion-language generation and understanding remains largely underexplored. Existing approaches often fine-tune large language models (LLMs) on paired motion-text data, which can result in catastrophic forgetting of linguistic capabilities due to the limited scale of available text-motion pairs. Furthermore, prior methods typically convert motion into discrete representations via quantization to integrate with language models, introducing substantial jitter artifacts from discrete tokenization. To address these challenges, we propose LLaMo, a unified framework that extends pretrained LLMs through a modality-specific Mixture-of-Transformers (MoT) architecture. This design inherently preserves the language understanding of the base model while enabling scalable multimodal adaptation. We encode human motion into a causal continuous latent space and maintain the next-token prediction paradigm in the decoder-only backbone through a lightweight flow-matching head, allowing for streaming motion generation in real-time (>30 FPS). Leveraging the comprehensive language understanding of pretrained LLMs and large-scale motion-text pretraining, our experiments demonstrate that LLaMo achieves high-fidelity text-to-motion generation and motion-to-text captioning in general settings, especially zero-shot motion generation, marking a significant step towards a general unified motion-language large model. Here are our project homepage: https://kunkun0w0.github.io/project/LLaMo/