UIO rchestra: Generating High-Fidelity Code from UI Designs with a Multi-agent System (2025)

Abstract: Recent advances in large language models (LLMs) have significantly improved automated code generation, enabling tools such as GitHub Copilot and CodeWhisperer to assist developers in a wide range of programming tasks. However, the translation of complex mobile UI designs into high-fidelity front-end code remains a challenging and underexplored area, especially as modern app interfaces become increasingly intricate. In this work, we propose UIOrchestra, a collaborative multi-agent system designed for the AppUI2Code task, which aims to reconstruct static single-page applications from design mockups. UIOrchestra integrates three specialized agents, layout description, code generation, and difference analysis agent that work collaboratively to address the limitations of single-model approaches. To facilitate robust evaluation, we introduce APPUI, the first benchmark dataset for AppUI2Code, constructed through a human-in-the-loop process to ensure data quality and coverage. Experimental results demonstrate that UIOrchestra outperforms existing methods in reconstructing complex app pages and highlight the necessity of multi-agent collaboration for this task. We hope our work will inspire further research on leveraging LLMs for front-end automation. The code and data will be released upon paper acceptance.

## introduction recently, large language models (llms) have significantly advanced the capabilities of various tasks. leveraging novel learning paradigms and massive code corpus, state-of-the-art llms even surpass human performance in tasks such as code generation from natural language instructions @xcite @xcite @xcite . these developments have led to the emergence of code assistance tools like github copilot foot_0 and amazon codewhisperer foot_1 , which provide developers with professional-level code completions and snippets from simple descriptions or partial code, improving development efficiency. however, there remain important and in-demand tasks where llms are less effective. the rise of mobile internet has led to a tremendous demand for app front-end development. modern app interfaces typically feature complex layouts and numerous components. front-end developers are required to accurately reproduce all elements specified in design mockups-including layout, hierarchy, and positioning-to construct static pages that faithfully reflect the original designs. while llms can assist with implementing interactive features, the initial process of static page construction remains tedious and time-consuming, resulting in high development costs. despite this demand, there has been limited research in this area. we refer to the challenging yet promising task of reconstructing static single-page apps from design mockups as appui2code. some studies have attempted to generate simple web pages by converting website screenshots or randomly constructed prototypes into html @xcite , a task commonly known as ui-to-code (ui2code) @xcite . while these approaches show the potential of llms in automating labor-intensive tasks, the lack of relevant data during pre-training often results in hallucinations, especially in fine-grained details. this limitation is further amplified when the focus shifts from simple web pages to complex modern app interfaces, where the high density of ui components increases the challenge for a single model. to address these challenges and enable efficient, high-quality static app page generation, we propose uiorchestra (as shown in figure 1 ), a multi-agent system for appui2code, which consists of three agents: a layout description agent (a ld ), a code generation agent (a cg ), and a difference analysis agent (a da ). a ld receives the ui design image and, guided by a few shot prompts, analyzes the layout in a row-by-row, divide-and-conquer manner, finally submitting a structural description of natural language. a cg takes both the layout description and the design image as input, constructs the overall code framework based on the layout description, fills in style details according to the design image, and render the generated code. a da compares the rendered page with the original design, identifies layout and style discrepancies, and iteratively feeds this information back to the code generation agent until the output is satisfactory. to evaluate uiorchestra, we examine existing benchmarks from prior work, which mainly focus on websites, finding them insufficient for assessing the reconstruction of complex app pages. to address this gap, we introduce appui, the first mobile-oriented benchmark for appui2code. unlike web front-end code, which can be easily collected at scale (e.g., common crawl, c4, webui), mobile app front-end data is much harder to obtain. even large companies are limited to their own products, leaving other widely used apps inaccessible. to overcome this, we employed human experts within uiorchestra, replacing a ld and a da in a human-in-the-loop process. experts pro-vided precise layout descriptions and difference analyses to a cg , producing high-quality initial code samples. these samples were further refined by experienced engineers to ensure fidelity to the original designs. this process resulted in a validation dataset of 1,101 samples. experimental results reveal the limitations of single-model approaches and underscore the necessity of multi-agent collaboration for complex tasks like appui2code. we hope future research will build on our work and, as model capabilities advance, further enhance the role of llms in frontend automation. in summary, our contributions are as follows: ## building static pages via uiorchestra appui2code is a challenging and complex task that requires large models to possess strong visual capabilities, accurately understand the hierarchical relationships and layout structures implied by the various components on a page, and at the same time, not overlook the stylistic details of each element, including color, position, font, etc. ultimately, all of this information must be faithfully converted into code. this entire process poses a significant challenge to the capabilities of llms. therefore, given the limitations of current models, it is a reasonable approach to decompose the complex task into a series of sub-tasks, each handled by a dedicated agent, allowing for division of labor and collaborative problem-solving. therefore, we propose uiorchestra, which aims to generate static frontend code from ui design images in a step-by-step manner through multi-agent collaboration. specifically, as shown in figure 2 , we introduce a layout description agent, denoted as a ld , and a difference analysis agent, denoted as a da , complementing the code generation agent to form a specialized multi-agent system for appui2code. collaboratively, these agents iteratively refine the quality of the code. the input reference screenshot is referenced as i 0 , while the front-end code produced by the code generation agent in the i th iteration and its corresponding rendered image are referenced as h i and i i , respectively. ## data curation with human-in-the-loop as discussed in section 1, existing benchmarks predominantly target web pages, which are often too simplistic to fully showcase the capabilities of uiorchestra. moreover, it is difficult to obtain front-end code for mobile applications, as such datasets cannot be easily gathered through web scraping. to address these challenges, we propose a human-in-the-loop approach that leverages uiorchestra and expert intervention to generate front-end code that faithfully replicates original design images, resulting in appui-the first benchmark for appui2code-as shown in figure 3 ming languages and frameworks by simply modifying the prompt requirements. in this work, we opt to generate front-end code in the react framework, as it is widely used as dsl in practice. for example, react native, which is extensively adopted, utilizes react during the development phase, and only maps to native components of the target system at the rendering stage on the app side through a dynamic framework. ## evaluation we conduct extensive benchmarking and human evaluation of various most recent powerful models and methods to explore the boundaries of their performance on our appui, including closed-source commercial models, open-source models, finetuned models, and our uiorchestra. ## conclusion our work highlights a missing capability in aiassisted programming: generating static page code from design images. to address this challenging task, we propose uiorchestra, a multi-agent collaborative framework that decomposes the problem into manageable steps. to rigorously evaluate uiorchestra, we introduce appui, the first benchmark for the appui2code domain. extensive experiments demonstrate the effectiveness of our approach. we hope this work will draw attention to this field, provide developers with more efficient tools, and promote further industry advancement.