S ketch2 C ode: Evaluating Vision-Language Models for Interactive Web Design Prototyping (2025)

Abstract: Sketches are a natural and accessible medium for UI designers to conceptualize early-stage ideas. However, existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration. To bridge this gap, we introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. Beyond end-to-end benchmarking, Sketch2Code supports interactive agent evaluation that mimics real-world design workflows, where a VLM-based agent iteratively refines its generations by communicating with a simulated user, either passively receiving feedback instructions or proactively asking clarification questions. We comprehensively analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs; even the most capable models struggle to accurately interpret sketches and formulate effective questions that lead to steady improvement. Nevertheless, a user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception, highlighting the need to develop more effective paradigms for multi-turn conversational assistants.

## introduction large language models (llms) have spurred a variety of applications on automating functional code implementations from natural language instructions @xcite @xcite . recent works such as @xcite and @xcite have started to explore possibilities of generating html code directly from full-fidelity web designs (e.g, mock-up screenshots) using vision language 1 code/data available on project page: https:// salt-nlp.github.io/sketch2code-project-page/ models (vlms), aiming to democratize frontend design for researchers, practitioners, and general users. however, the screenshot-to-code setting is inconvenient as providing detailed graphical designs for the desired user interface (ui) is timeconsuming and sometimes requires professional tools with a steep learning curve. on the other hand, sketching is a low-fidelity, accessible, and plentiful tool that is much easier to learn and implement (sturdee and @xcite . despite their low fidelity, sketches are commonly used to ideate, communicate, and visualize design concepts, often serving as the earliest yet vitally important step of ui designs @xcite @xcite . transforming sketches to code used to be implemented in a pipeline fashion that involves pattern and object recognition @xcite @xcite . however, recent development of general-purpose vlms @xcite @xcite began to shift this paradigm by enabling such transformation end-to-end. in this paper, we present sketch2code, a first-of-its-kind framework to access vlms' capability of implementing web ui from user sketches, where we (1) collected 731 high-quality sketches from 484 real-world webpages through crowd workers based on @xcite , ( 2 ) assessed vlms' performance on directly transforming sketches to code, and (3) designed a multi-turn, interactive framework to benchmark vlms on sketch2code using llm simulated users, unlike prior works @xcite that focused solely on single-turn generations. real-world web design is an iterative process where initial concepts undergo multiple revisions based on continuous feedback and clarifications @xcite . especially while using sketches, which are low-fidelity representations, it is impossible to figure out specific details, such as stylistic information, without additional input from the user. to mirror realistic design workflows and study how well vlms can interact with humans, our framework further introduces two multi-turn evaluation scenarios between a sketch2code agent and a simulated user: (1) the sketch2code agent follows feedback from the simulated user (feedback following) and ( 2 ) the sketch2code agent proactively asks the simulated user questions for design details and clarification (question asking). to this end, our framework assesses not only the ability of models to generate initial implementations based on abstract inputs but also their capacity to adapt and evolve these implementations in response to user feedback. since these are two of the most common communication patterns between human collaborators, our framework allows the simulated user to be seamlessly replaced by real users in a real-world deployment. human annotations reveal that the simulated user provides faithful and meaningful feedback 83.3% of the time and answers 86.7% of the questions accurately. we conducted a comprehensive analysis of ten models: gpt-4o, gpt-4o mini, gemini 1.5 pro, gemini 1.5 flash, claude 3.5 sonnet, claude 3 opus/sonnet/haiku, llava-1. results indicate that inferring the correct layout structures from rudimentary sketch designs is challenging for vlms in a single turn @xcite . while commercial models perform reasonably well in following user feedback-achieving performance improvements of up to 7.1% in visual similarity @xcite ) and 2.7% in iou-based @xcite layout similarity within five rounds of interaction-all vlms struggle to formulate meaningful questions about the sketches and fail to reliably enhance their performance across multiple rounds in the question-asking scenario ( § 4.4). to understand the utility of our sketch-based, multiturn framework, we conducted a user study involving eight ui/ux practitioners while all participants recognized the usefulness of our framework. furthermore, it reveals that the question-asking mode is significantly preferred over the more traditional feedback-following mode despite current suboptimal performances. specifically, users prefer the agent to proactively undertake more of the cognitive workload and guide the design choices via a series of targeted questions. although questions asked by agents are often found ineffective, replacing model-generated questions with questions asked by human experts leads to a significant improvement with each question ( § 5). our findings highlight a critical gap between user expectations and current model capabilities, necessitating further research into human-ai collaboration and the capability of more proactive interactions. frontend ui code generation. nguyen and csallner (2015) pioneered reverse engineering mobile uis using ocr and computer vision techniques to generate code. pix2code @xcite introduced an end-to-end ui-to-code system leveraging cnns and rnns, but faced challenges with complex visual encoding and text decoding. aşıroglu et al. (2019) ## experiments 4.1 prompting methods we evaluated the agents using two prompting methods: direct prompting and text-augmented prompting. in direct prompting, the agent is provided with a sketch design only and is asked to generate an html prototype without access to additional information. in text-augmented prompting, we augment the agent with all text content extracted from the reference webpage as part of the initial user prompt, following @xcite . the exact prompts used and additional experiment details are available in appendix b. preliminary results show that text-augmented prompting is a more realistic setup and yields the most stable outputs across multiple interaction turns. hence, we use text-augmented prompting as the starting point for all multi-turn experiments. ## user study and analysis to better understand the importance of sketching in the ui/ux development cycle, and the potential use cases and implications of a sketch2code agent, we conducted a user study by interviewing eight ui/ux experts recruited from upwork foot_4 . all experts agreed that low-fidelity sketches play a substantial role in modern ui/ux development. furthermore, they found that a sketch2code agent would significantly benefit their work. a sketch2code agent can help users quickly flesh out early-stage ideas and break the communication barrier between clients and designers. more interestingly, seven out of eight participants showed strong preference towards the question-asking agent. the interviewees expressed that they needed to specify every design detail to an agent that passively follows user instructions. in contrast, a questionasking agent can take over most of the cognitive workload, and the user only needs to focus on the parts being asked. three of the experts pointed out that the agent can proactively "guide" the user through certain design decisions & choices via a series of targeted questions, so the user do not have to figure out every single detail themselves. moreover, the participants mentioned that it is difficult to select visual components and specify visual information with natural language feedback-/answers alone. when communicating design ideas in real life, people can simply point to specific visual components by mouse or by finger instead of using words, which is faster, easier, and more reliable. detailed findings are available in appendix e. moreover, we invited 5 ui/ux experts who participated in the user studies to (1) provide feedback on 100 generated designs from three models (gpt-4o, claude-3-opus, and gemini-1.5-pro) and ( 2 ) ask questions on 100 sketches. we further compare the difference between ai-generated and humanwritten feedback/questions below 6 . simulated user feedback achieved comparable performance as feedback from real human experts: on average, each feedback from the simulated user improves the agents' performance in visual similarity by 1.41% and layout similarity by 1.05%, whereas each feedback from a human expert improves the visual similarity by 1.62% and layout similarity by 0.98%. we developed a taxon-omy and summarized the user feedback into seven categories (detailed taxonomy and statistics are available in appendix f.). interestingly, human experts provide more feedback on colors and styling, while simulated users focus more on texts and general comments about the webpage. furthermore, we evaluated the effectiveness of each type of simulated feedback in appendix the effectiveness of questions asked by models lags far behind that of experts. initially, the models' performance on average decreases by 1.12% in visual similarity and improves by 0.74% in layout similarity after asking each question. however, by replacing the agent-generated questions with questions written by human experts, the models were able to improve their visual similarity by 0.58% and layout similarity by 1.49% with each question. based on a taxonomy of the nine most common types of questions (the full taxonomy and further details in appendix f.), we found that human experts often ask about the general styling and layout of the webpage, whereas the agents often focus on the specific placements of textual contents or ask irrelevant questions. as shown in appendix ## conclusion and future work in this work, we introduced sketch2code, a novel interactive evaluation framework that assesses vision language models' (vlms) capability for multi-turn front-end ui/ux automation. we proposed two interaction paradigms: feedback following and question asking. our evaluations revealed that while modern commercial vlms perform reasonably well in feedback following-improving visual and layout similarities by 3% and 1.8% over five interaction rounds-they struggle with asking meaningful questions. however, a user study with eight ui/ux practitioners showed a user preference for the question-asking paradigm, as it allows the agent to take on more cognitive responsibilities, highlighting a gap between user expectations and current model capabilities. we outline future research directions inspired by sketch2code: (i) training open-source models for multi-turn ui generations. this can be challenging due to the long contexts and multi-modality. to facilitate large-scale training, we present an automated pipeline for generating realistic synthetic sketches at scale (see appendix h). (ii) developing agentic frameworks that are more capable of cognitive reasoning and proactively guiding users through multi-turn design workflows, instead of passively following instructions. (iii) creating end-to-end ui/ux ai applications to enhance designer productivity and make ui designs more accessible to non-experts.