ModularAgent：面向多模态大语言模型与世界模型联合优化的任务感知模块化框架
ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models
把 MLLM 的语义表征注入世界模型潜空间、再用世界模型生成的稠密文本条件奖励反向精炼 MLLM 语义空间的双向耦合机制，可为视觉研究者提供一种将视觉动态预测与语言语义先验联合训练的可复用范式，尤其适用于需要多任务与跨环境泛化的具身视觉策略学习。
原文依据 a forward path that injects MLLM representations into the WMs latent space for semantically guided imagination
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论构建通用具身智能体需要一个能够解读多模态目标、建模环境动态并在多样化的真实世界任务中执行可靠动作的统一系统。多模态大语言模型（MLLM）提供强大的语义先验与跨模态泛化能力，而世界模型（WM）则为预测与控制提供可操作的潜变量动态。二者的结合为开放式具身智能带来了希望，但也引入了两个关键挑战：（1）在 MLLM 的语义意图与世界模型潜空间中的动态状态表征之间建立紧密耦合；（2）实现任务感知的自适应性，以支持多任务学习与跨环境泛化。为解决上述局限，我们提出 ModularAgent，一个任务感知的动态联合框架，可实现 MLLM 与世界模型之间的双向耦合。ModularAgent 建立了两条互补的路径：前向路径将 MLLM 表征注入世界模型的潜空间，以实现语义引导的想象；反向路径则由世界模型生成的反馈通过稠密的文本条件奖励来精炼 MLLM 的语义空间。这种双向交互通过三个协同组件实现：任务感知动态联合学习、任务感知行为学习以及 MLLM-WM 联合优化，三者共同协调语义推理与动态预测。在多任务与跨环境设置下的大量实验表明，该方法相比最优基线具有更优的稳定性与泛化能力，朝着开放式具身学习迈出了一步。
英文原文摘要 1623 字符
Building generalist embodied agents requires a unified system that can interpret multimodal goals, model environment dynamics, and execute reliable actions across diverse real-world tasks. Multimodal large language models (MLLMs) offer strong semantic priors and cross-modal generalization, while world models (WMs) provide actionable latent dynamics for prediction and control. Their combination holds promise for open-ended embodied intelligence, yet introduces two key challenges: (1) establishing a tight coupling between the semantic intent from MLLMs and the dynamic state representations within the WM's latent space, and (2) achieving task-aware adaptability that supports multi-task learning and cross-environment generalization. To address these limitations, we propose ModularAgent, a task-aware dynamic joint framework that enables bidirectional coupling between MLLMs and WMs. ModularAgent establishes two complementary pathways: a forward path that injects MLLM representations into the WMs latent space for semantically guided imagination, and a backward path where WM-generated feedback refines the MLLMs semantic space via dense text-conditioned rewards. This bidirectional interaction is realized through three synergistic components: Task-Aware Dynamic Joint Learning, Task-Aware Behavior Learning, and MLLM-WM Joint Optimization, which together harmonize semantic reasoning and dynamic prediction. Extensive experiments across multi-task and cross-environment settings demonstrate superior stability and generalization over state-of-the-art baselines, marking a step toward open-ended embodied learning.