Widget2Code：通过多模态大语言模型从视觉控件到 UI 代码
Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs
把「从图像生成结构化代码」当作多模态大模型的细粒度视觉理解任务：用图标检索与 UI 组合模块把原子组件拼成完整布局，可作为视觉到结构化序列生成（而非自由文本）的可复用范式，并提供图像级 UI 重建的细粒度评测协议。
原文依据 we develop a baseline that jointly advances perceptual understanding and structured code generation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论用户界面到代码（UI2Code）旨在生成能够忠实重建给定输入 UI 的可执行代码。已有工作大多聚焦于网页与移动端屏幕，对应用控件（widget）的探索仍不充分。与具有丰富层级上下文的网页或移动端 UI 不同，控件是紧凑、无上下文的微型界面，在严格的空间约束下通过密集布局与图标来概括关键信息。此外，网页或移动端 UI 广泛存在（图像，代码）配对数据，而控件设计属于专有资产且缺乏可获取的标记。我们将这一设定形式化为控件到代码（Widget2Code），并引入一个仅含图像的控件基准，配备细粒度、多维度的评测指标。基准测试表明，尽管通用多模态大语言模型（MLLM）优于专门的 UI2Code 方法，它们仍会生成不可靠且视觉上不一致的代码。为解决这些局限，我们开发了一个同时推进感知理解与结构化代码生成的基线。在感知层面，我们遵循控件设计原则，将原子组件组装为完整布局，并配备图标检索与 UI 组合模块。在系统层面，我们设计了端到端基础设施 WidgetFactory，其中包含与框架无关、为控件定制的领域特定语言（WidgetDSL），以及可将其翻译为多种前端实现（如 React、HTML）的编译器。一个自适应渲染模块进一步精修空间尺寸以满足紧凑性约束。这些贡献共同显著提升了视觉保真度，为未来的 Widget2Code 研究建立了强基线（baseline）与统一基础设施。
英文原文摘要 1733 字符
User interface to code (UI2Code) aims to generate executable code that can faithfully reconstruct a given input UI. Prior work focuses largely on web pages and mobile screens, leaving app widgets underexplored. Unlike web or mobile UIs with rich hierarchical context, widgets are compact, context-free micro-interfaces that summarize key information through dense layouts and iconography under strict spatial constraints. Moreover, while (image, code) pairs are widely available for web or mobile UIs, widget designs are proprietary and lack accessible markup. We formalize this setting as the Widget-to-Code (Widget2Code) and introduce an image-only widget benchmark with fine-grained, multi-dimensional evaluation metrics. Benchmarking shows that although generalized multimodal large language models (MLLMs) outperform specialized UI2Code methods, they still produce unreliable and visually inconsistent code. To address these limitations, we develop a baseline that jointly advances perceptual understanding and structured code generation. At the perceptual level, we follow widget design principles to assemble atomic components into complete layouts, equipped with icon retrieval and UI composition modules. At the system level, we design an end-to-end infrastructure, WidgetFactory, which includes a framework-agnostic widget-tailored domain-specific language (WidgetDSL) and a compiler that translates it into multiple front-end implementations (e.g., React, HTML). An adaptive rendering module further refines spatial dimensions to satisfy compactness constraints. Together, these contributions substantially enhance visual fidelity, establishing a strong baseline and unified infrastructure for future Widget2Code research.