D esign2 C ode: Benchmarking Multimodal Code Generation for Automated Front-End Engineering (2025)

Abstract: Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development in which multimodal large language models (MLLMs) directly convert visual designs into code implementations. In this work, we construct Design2Code – the first real-world benchmark for this task. Specifically, we manually curate 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations to validate the performance ranking. To rigorously benchmark MLLMs, we test various multimodal prompting methods on frontier models such as GPT-4o, GPT-4V, Gemini, and Claude. Our fine-grained break-down metrics indicate that models mostly lag in recalling visual elements from the input webpages and generating correct layout designs.

## introduction implementing visual designs of websites into functional code is a challenging task as it requires understanding visual elements and their layouts and then translating them into structured code. such dependencies on sophisticated skills have prevented many laypeople from building their own web applications, even when they have concrete ideas for what to build. furthermore, the requirement for domain expertise complicates the whole webpage production pipeline, requiring collaboration among people with different skill sets and potentially causing discrepancies between the intended design and actual implementation. effective automatic generation of functional code from visual designs has * equal contribution. the potential to democratize the development of front-end web applications @xcite , allowing non-experts to build applications easily and quickly. while code generation from natural language instructions has advanced rapidly in recent years @xcite @xcite , generating code implementation from user interface (ui) design has not received much attention due to a wide range of challenges, such as diversity in visual and text signals on the user interface and the vast search space in the resulting code. @xcite made a notable attempt back in 2017 with cnn and rnn models on a narrow set of simplistic ui designs. over the years, despite many follow-up attempts along this quest @xcite , they are all constrained to simplistic or synthetic examples with a narrow set of layout designs, hardly useful for real-world front-end development applications. until recently, the development of multimodal llms has entered a new era where largescale pretrained models can process both visual and text input and generate text output for various visually grounded tasks, with representative examples like flamingo @xcite , gpt-4v (ope-nai, 2023), and gemini @xcite . such advancement has unlocked a brand new paradigm for this long-standing unsolved task: give the user's website design as an image input to the system to obtain the full code implementation that can render into the desired webpage in an end-to-end manner. to systematically and rigorously benchmark current mllms on this task, we construct the firstever real-world benchmark design2code. to best reflect realistic use cases, we use realworld webpages (examples in figure 1 ) in the wild as our test examples rather than synthetically generated ones as in prior works @xcite . we scrape webpages in the c4 @xcite validation set and per-figure 1 : examples from the prior websight v0.1 dataset (first row) and our new design2code benchmark (last two rows). we use real-world webpages for benchmarking to ensure they are realistic, diverse, and complex, while websight uses synthetically generated simple webpages for scalability. we then investigate how current mllms perform on this task, by testing a variety of prompting methods, including our text-augmented prompting that complements visual input with extracted text elements from the webpage to reduce the load on ocr, as well as a self-revision prompting method that asks the model to compare its previous generation and the input webpage for self-improvement. we see significant improvement from text-augmented prompting on most models, while few can self-revise their generation. additionally, we also construct design2code-hard, a separate set of 80 hard examples, to compare stateof-the-art commercial models (gpt-4o, claude 3.5 sonnet) on these challenging cases. ## the design2code benchmark in this section, we describe the curation and processing of our benchmark data. we first scrape all website links in the c4 @xcite validation set. we then embed all css code into the html file to obtain one single code implementation file for each webpage. this results in a total of 127.9k webpages, which we perform further filtering and processing as described below. ## benchmarking: prompting and finetuning we benchmark a variety of models and methods to compare their performance on our benchmark, including commercial api models, open-source models, and finetuned models. ## conclusion this work introduced the design2code(-hard) benchmark consisting of diverse and challenging real-world webpages as test examples. we developed comprehensive automatic metrics and conducted human evaluations to compare various multimodal llms. we showed that state-of-the-art models can generate well-formed websites on some easy examples but still struggle with more complex examples, while open-source models are far behind commercial models, leaving ample room for future improvement. for future work, a promising direction is to generate uis that support dynamic user interactions.