底层视觉与图像复原 · CVPR2026 ·2026.06.03

超越串行工具:基于动态多专家融合的统一视觉语言模型智能体系统用于摄影后期处理

Beyond Sequential Tools: A Unified VLM Agent System for Photographic Post-Processing via Dynamic Multi-Expert Fusion

Honglin Xiong、Chenjie Zhu、Jianbiao Ding、Zixuan Ni、Wei Li、Zhenpeng Mi 等 7 人

与你有关的部分

把 VLM 当作调度器,为多个只改 K/V 矩阵的专家 LoRA 动态分配权重,并一次性合并进扩散骨干网络完成单步复原,这个「多专家 LoRA 融合 + 单次前向」的范式可直接迁移到去噪、去模糊、低光照增强等耦合退化场景。

原文依据 these experts adapt only Key (K) and Value (V) matrices and are simultaneously merged into a pretrained diffusion backbone
可迁移到 图像复原图像增强扩散模型微调

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

真实世界中的图像复原面临复杂且相互耦合的退化问题。现有的「一体化」模型往往缺乏泛化能力,而智能体系统则受困于低效的串行工具调用。我们提出一种由视觉语言模型引导的单步框架,用于通用的摄影后期处理。该系统采用视觉语言模型(VLM)作为调度器,执行细致的意图理解与退化分析,并为一组专门的专家低秩适配模块动态分配权重。为确保更优的可组合性,这些专家仅调整键(K)与值(V)矩阵,并被同时合并进一个预训练的扩散骨干网络,以实现协同的单次前向复原。此外,我们引入一个通过直接偏好优化(DPO)训练的轻量分支,以保证感知最优的权重分配。本方法在多种合成与真实世界数据集上均取得最优性能。关键在于,它在真实世界数据上展现出显著的零样本泛化能力,且无需额外微调。

英文原文摘要 1073 字符

Real-world image restoration is challenged by complex, coupled degradations. Existing "all-in-one" models often lack generalization, while agentic systems suffer from inefficient sequential tool invocation. We propose a VLM-guided one-shot framework for universal photographic post-processing. Our system employs a Vision-Language Model (VLM) as an orchestrator to perform nuanced intent understanding and degradation analysis, dynamically allocating weights to a suite of specialized expert LoRA modules. To ensure superior composability, these experts adapt only Key (K) and Value (V) matrices and are simultaneously merged into a pretrained diffusion backbone for synergistic, single-pass restoration. Furthermore, we introduce a lightweight branch trained via Direct Preference Optimization (DPO) to ensure perceptually optimal weight allocation. Our method achieves state-of-the-art performance across diverse synthetic and real-world datasets. Crucially, it demonstrates remarkable zero-shot generalization on authentic real-world data without additional fine-tuning.

标签

会议 CVPR2026
页码 41521-41530
Bibkey
来源 CVF Open Access