用于图像复原的混合智能体
Hybrid Agents for Image Restoration
将多模态大语言模型与轻量级LLM结合为快慢双智能体，通过指令微调与上下文学习实现图像复原任务的自动工具调用与混合退化去除，为构建可交互、自适应的复原系统提供了可复用的智能体框架。
原文依据 we present HybridAgent, intending to incorporate multiple restoration modes into a unified image restoration model
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有的图像复原研究通常分别关注特定任务或通用模式，依赖用户进行模式选择，且缺乏多种特定任务/通用复原模式之间的协作。这导致非专业用户交互不足，并限制了他们在复杂现实应用中的复原能力。在本工作中，我们提出HybridAgent，旨在将多种复原模式整合到一个统一的图像复原模型中，并通过我们提出的混合智能体实现智能且高效的用户交互。具体而言，我们提出了快速、慢速和反馈复原智能体的混合规则。其中，慢速复原智能体利用我们提出的指令微调数据集优化强大的多模态大语言模型（MLLM），以在用户提示模糊时识别图像中的退化，并相应地调用适当的复原工具。快速复原智能体基于轻量级大语言模型（LLM）通过上下文学习设计，用于理解简单清晰要求的用户提示，从而避免MLLM不必要的时间/资源开销。此外，我们为HybridAgents引入了混合失真去除模式，这在以往基于智能体的工作中至关重要但未被关注。它可以有效防止逐步图像复原的错误传播，并大幅提高智能体系统的效率。我们通过合成和真实世界的图像复原任务验证了HybridAgent的有效性。
英文原文摘要 1554 字符
Existing Image Restoration (IR) studies typically focus on task-specific or universal modes individually, relying on the mode selection of users and lacking cooperation between multiple task-specific/universal restoration modes. This leads to insufficient interaction for unprofessional users and limits their restoration capability for complicated real-world applications. In this work, we present HybridAgent, intending to incorporate multiple restoration modes into a unified image restoration model and achieve intelligent and efficient user interaction through our proposed hybrid agents. Concretely, we propose the hybrid rule of fast, slow, and feedback restoration agents. Here, the slow restoration agent optimizes the powerful multimodal large language model (MLLM) with our proposed instruction-tuning dataset to identify degradations within images with ambiguous user prompts and invokes proper restoration tools accordingly. The fast restoration agent is designed based on a lightweight large language model (LLM) via in-context learning to understand the user prompts with simple and clear requirements, which can obviate the unnecessary time/resource costs of MLLM. Moreover, we introduce the mixed distortion removal mode for our HybridAgents, which is crucial but not concerned in previous agent-based works. It can effectively prevent the error propagation of step-by-step image restoration and largely improve the efficiency of the agent system. We validate the effectiveness of HybridAgent with both synthetic and real-world IR tasks.