ROSE：旋转你的大语言模型使其看见
ROSE: Rotate Your Large Language Model to See
提出用旋转矩阵把视觉语义直接作用到 LLM 预训练参数上的参数空间注入方式，替代把视觉 token 拼进输入序列的做法；做视觉-语言对齐、想降低多模态上下文长度与推理成本的工作可以直接借鉴这一注入接口。
原文依据 Visual semantics are encoded as rotation matrices and applied directly to the pretrained parameters.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期多模态大语言模型（MLLM）的进展在融合视觉与语言理解方面取得了令人瞩目的成果。然而，现有大多数 MLLM 将视觉信息注入大语言模型（LLM）的输入空间，这显著增加了上下文长度与计算开销，同时往往迫使 LLM 在视觉主导的多模态序列上优化，从而破坏预训练的语言先验。在本工作中，我们提出一种基于旋转的视觉注入范式，将视觉信息与 LLM 的参数空间对齐。视觉语义被编码为旋转矩阵并直接作用于预训练参数。这种参数空间注入无需长输入序列，从而避免了输入空间注入固有的二次计算开销。此外，它通过保持预训练参数的内在几何结构来保留 LLM 的语言能力。基于该范式，我们构建了 ROSE，一个 7B 的 MLLM，以出色的计算效率实现了细粒度的视觉-语言对齐。在 12 个多模态基准上的大量实验表明，ROSE 相比领先模型取得了更优或相当的性能。在精度相当的情况下，ROSE 相对 Qwen2.5-VL-7B 将浮点运算量降低 80.7%、推理延迟降低 56.4%，验证了其有效性与可扩展性。代码与模型权重将公开发布。
英文原文摘要 1490 字符
Recent advances in multimodal large language models (MLLMs) have shown impressive progress in integrating visual and linguistic understanding. However, most existing MLLMs inject visual information into the input space of large language models (LLMs), which substantially increases context length and computational overhead, while often disrupting pretrained linguistic priors by forcing the LLM to optimize on vision-dominant multimodal sequences. In this work, we propose a rotation-based vision injection paradigm that aligns visual information with the parameter space of LLMs. Visual semantics are encoded as rotation matrices and applied directly to the pretrained parameters. This parameter-space injection eliminates the need for long input sequences, thus avoiding the quadratic computational overhead inherent in input-space injection. Besides, it preserves the linguistic competence of the LLM by maintaining the intrinsic geometric structure of the pretrained parameters. Building upon this paradigm, we develop ROSE, a 7B MLLM that achieves fine-grained vision-language alignment with remarkable computational efficiency. Extensive experiments across 12 multimodal benchmarks show that ROSE delivers superior or competitive performance compared with leading models.At comparable accuracy, ROSE reduces FLOPs by 80.7% and inference latency by 56.4% relative to Qwen2.5-VL-7B, demonstrating the effectiveness and scalability. The code and model weights will be publicly released.