ReFAct：赋予多模态网络智能体视觉与上下文聚焦能力
ReFAct: Empowering Multimodal Web Agents with Visual and Context Focusing
提出 Grounding 工具进行主动视觉感知以动态过滤视觉信息，并设计基于外部记忆的 Defocus/Refocus 操作调节多模态上下文信息密度，可复用于视觉语言模型在噪声环境下的鲁棒推理。
原文依据 ReFAct employs a Grounding tool for active visual perception to dynamically filter information.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态网络搜索智能体通过融合来自多种模态（如文本和视觉）的信息，从互联网上迭代检索以解决复杂用户查询，展现出具有实际价值的能力。然而，视觉模态容易面临信息过载问题，其中包含的噪声——如无关的背景细节或复杂结构——会干扰模型的注意力，将其操作焦点误导至错误路径。为应对上述挑战，我们提出 ReFAct（推理、聚焦与行动），一种新颖的框架，赋予智能体主动管理其跨模态上下文的能力。这使得智能体能够调整其操作焦点，从而减轻噪声对多模态网络搜索智能体的影响。具体而言，ReFAct 采用视觉定位工具进行主动视觉感知以动态过滤信息。我们还设计了基于外部记忆的散焦/重新聚焦操作，用于选择性信息保留，进一步调节多模态上下文中的信息密度。最终，这确保智能体在问题解决过程中保持聚焦。为评估并增强智能体在复杂且含噪多模态上下文中的能力，我们首先提出了一种构建具有灵活复杂度数据集的流程。我们引入了一个新的开源基准：GroundedVQA。最后，我们在 GroundedVQA 及其他广泛使用的基准上通过实验证明了所提方法的有效性。
英文原文摘要 1486 字符
Multimodal Web Search Agents demonstrate a practically valuable capability by fusing information from diverse modalities (e.g., text and vision), retrieved iteratively from the internet, to address complex user queries. However, the visual modality is prone to information overload, and the noise contained within it--such as irrelevant background details or complex structures--can disrupt the model's attention, misdirecting its operational focus toward an erroneous path. To address the aforementioned challenge, we propose ReFAct (Reasoning, Focusing, and Acting), a novel framework that empowers the agent to actively manage its cross-modal context. This allows the agent to adjust its operational focus, thereby mitigating the impact of noise on multimodal Web Search Agents. Specifically, ReFAct employs a Grounding tool for active visual perception to dynamically filter information. We also design external memory-based Defocus/Refocus operations for selective information retention, further modulating information density within the multimodal context. Ultimately, this ensures the agent maintains focus during problem-solving. To evaluate and enhance agent capabilities in complex and noisy multimodal contexts, we first propose a pipeline for constructing datasets with flexible complexity. We introduce a new open-source benchmark: GroundedVQA. Finally, we experimentally demonstrate the effectiveness of our proposed method on GroundedVQA and other widely-used benchmarks.