生成模型与扩散 · CVPR2026 ·2026.06.03

Hint2Gen:通过代码结构化提示连接理解与生成

Hint2Gen: Bridging Understanding and Generation via Code-structured Hints

Yuanpeng Tu、Yunpeng Chen、Xi Chen、Liang Li、Hengshuang Zhao

与你有关的部分

把 SVG/HTML 这类代码结构化提示作为图像平面上的显式条件输入,接到 FLUX.1 Kontext 这类扩散编辑模型上,为「推理密集型生成」提供了一条可复用的条件注入路径;Reason2Gen 基准也可直接用于评测生成模型的推理能力。

原文依据 we propose using code-structured visual hints (i.e., SVG/HTML) overlays that explicitly encode reasoning steps directly on the image plane
可迁移到 reasoning-aware image generationinstruction-based image editingcontrollable generation

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

近期的统一模型在生成高质量图像方面取得了显著进展,但在推理密集型任务(即求解迷宫、拼装七巧板)上却始终表现不佳。有趣的是,我们发现视觉语言模型(VLM)与大语言模型(LLM)能够准确求解这些任务,却无法生成相应的图像,因为它们缺乏结构化的视觉输出接口。这表明核心瓶颈并非推理能力,而是缺乏一个将高层推理转化为精确视觉输出的结构化接口。为弥合这一差距,我们提出使用代码结构化的视觉提示(即 SVG/HTML)叠加层,在图像平面上直接显式编码推理步骤。据此,我们开发了一套自动数据构建流程,可为现有数据集生成高质量的代码结构化提示,并基于 FLUX.1 Kontext 训练了一个名为 Hint2Gen 的统一模型,使其以这类提示为条件进行生成。此外,为全面评估我们方法的有效性,我们提出了 Reason2Gen,这是一个包含 4,000 个样本、覆盖 7 个核心维度下 20 个类别的基准,涵盖路径连通性、空间拼装等。大量实验表明,即使仅将这类提示作为额外输入提供——无需任何重新训练——也能提升其性能。而我们的模型在全部维度上的推理感知生成与编辑任务中均显著优于所有领先的开源/闭源方法。

英文原文摘要 1516 字符

Recent unified models have made remarkable strides in generating high-quality images, yet they consistently fail on reasoning-intensive tasks, i.e., solving mazes, assembling tangrams. Intriguingly, we find that vision-language models (VLMs) and large language models (LLMs) can accurately solve these tasks, but cannot generate the corresponding images because they lack a structured visual output interface. This reveals that the core bottleneck is not reasoning capacity, but the lack of a structured interface to translate high-level reasoning into precise visual output. To bridge this gap, we propose using code-structured visual hints (i.e., SVG/HTML) overlays that explicitly encode reasoning steps directly on the image plane. Accordingly, we develop an automatic data construction pipeline that can generate high-quality code-structured hints for existing datasets and train a unified model called Hint2Gen based on FLUX.1 Kontext to condition its generation on such hints. Furthermore, to comprehensively evaluate the effectiveness of our approach, we introduce Reason2Gen, a benchmark comprising 4,000 samples spanning 20 categories across 7 core dimensions, including path connectivity, spatial assembly, etc. Extensive experiments demonstrate that even simply providing such hints as extra inputs--without any retraining--boosts their performance. And our model significantly outperforms all leading open-source/closed-source methods on reasoning-aware generation and editing across all the dimensions.

标签

会议 CVPR2026
页码 36593-36603
Bibkey
来源 CVF Open Access