CodePercept：面向多模态大模型的代码接地视觉STEM感知
CodePercept: Code-Grounded Visual STEM Perception for MLLMs
将可执行代码作为视觉感知的监督信号，通过图像到代码的生成任务迫使模型进行精确的视觉理解，为评估和增强多模态大模型的细粒度感知能力提供了新范式。
原文依据 establishing code as a powerful perceptual medium--executable code provides precise semantics that naturally align with the structured nature of STEM visuals
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当多模态大模型在科学、技术、工程和数学（STEM）视觉推理中失败时，一个根本问题浮现：这是由于感知缺陷还是推理限制？通过独立缩放感知和推理组件的系统缩放分析，我们揭示了一个关键洞见：缩放感知始终优于缩放推理。这表明感知是限制当前STEM视觉推理的真正杠杆。受此洞见启发，我们的工作聚焦于通过将代码确立为强大的感知媒介来系统增强多模态大模型的感知能力——可执行代码提供了精确的语义，自然契合STEM视觉的结构化特性。具体而言，我们构建了ICC-1M，一个包含100万图像-描述-代码三元组的大规模数据集，通过两种互补方法实现这种代码即感知范式：（1）代码接地的描述生成将可执行代码视为图像描述的真值，消除了现有知识蒸馏方法中固有的幻觉；（2）STEM图像到代码翻译提示模型生成重建代码，缓解了自然语言在感知增强中的模糊性。为验证该范式，我们进一步引入了STEM2Code-Eval，一个直接评估STEM领域视觉感知的新基准。与依赖问题解决准确率作为代理（仅衡量问题相关理解）的现有工作不同，我们的基准要求通过可执行代码生成进行图像重建来全面理解视觉，提供确定性和可验证的评估。代码见 https://github.com/TongkunGuan/Qwen-CodePercept。
英文原文摘要 1757 字符
When MLLMs fail at Science, Technology, Engineering, and Mathematics (STEM) visual reasoning, a fundamental question arises: is it due to perceptual deficiencies or reasoning limitations? Through systematic scaling analysis that independently scales perception and reasoning components, we uncover a critical insight: scaling perception consistently outperforms scaling reasoning. This reveals perception as the true lever limiting current STEM visual reasoning. Motivated by this insight, our work focuses on systematically enhancing the perception capabilities of MLLMs by establishing code as a powerful perceptual medium--executable code provides precise semantics that naturally align with the structured nature of STEM visuals. Specifically, we construct ICC-1M, a large-scale dataset comprising 1M Image-Caption-Code triplets that materializes this code-as-perception paradigm through two complementary approaches: (1) Code-Grounded Caption Generation treats executable code as ground truth for image captions, eliminating the hallucinations inherent in existing knowledge distillation methods; (2) STEM Image-to-Code Translation prompts models to generate reconstruction code, mitigating the ambiguity of natural language for perception enhancement. To validate this paradigm, we further introduce STEM2Code-Eval, a novel benchmark that directly evaluates visual perception in STEM domains. Unlike existing work relying on problem-solving accuracy as a proxy that only measures problem-relevant understanding, our benchmark requires comprehensive visual comprehension through executable code generation for image reconstruction, providing deterministic and verifiable assessment. Code is available at https://github.com/TongkunGuan/Qwen-CodePercept.