UIC oder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback (2024)

Abstract: Many large language models (LLMs) struggle to consistently generate UI code that compiles and produces visually relevant designs. Existing approaches to improve generation rely either on expensive human feedback or distilling a proprietary model. In this paper, we explore the use of automated feedback (compilers and multi-modal models) to guide LLMs to generate high-quality UI code. Our method starts with an existing LLM and iteratively produces improved models by self-generating a large synthetic dataset using an original model, applying automated tools to aggressively filter, score, and de-duplicate the data into a refined higher quality dataset, and producing a new LLM by finetuning the original on the refined dataset.We applied our approach to several open-source LLMs and compared the resulting performance to baseline models with both automated metrics and human preferences.Our results show the resulting models outperform all other downloadable baselines and approach the performance of larger proprietary models.

## introduction a well-known survey estimates that user interface (ui) code comprises roughly half of a user-facing gui application's code @xcite . however, writing ui code has traditionally been a difficult and time-consuming process that requires substantial effort and expertise. large language models (llms) present a promising solution since they are trained on large amounts of natural language text and code, which allows them to relate high-level user specifications to concrete code implementations. following a large-scale unsupervised "pretraining" phase, the model is finetuned to perform specific tasks, such as generating code, based on natural language directives in a process called instruction-tuning. through this process, llms have been trained to be proficient in conversation, develop reasoning abilities, and even use external tools @xcite . nevertheless, it is still difficult for many llms to reliably generate syntactically-correct, welldesigned code for uis, which suggests that it is an understudied problem. most of the examples found in crawled web pages and repositories are not self-contained or are of low quality @xcite . even in curated or manually authored finetuning datasets, examples of ui code are extremely rare, in some cases making up less than one percent of the overall examples in code datasets @xcite . besides being a "low-resource" languages @xcite , ui code and toolkits are also distinguished by their programming styles, such as frequent use of functional and reactive conventions. in this paper, we describe an automated method for training llms to generate ui code from textual descriptions. we specifically focus on training models to implement uis using swiftui, the official framework for popular apple platforms, though our method would likely generalize to other languages and ui toolkits. instead of relying on additional external data, our approach finetunes llms to generate improved ui code entirely from their own previous outputs. we first prompt an existing llm to generate a large synthetic dataset of swiftui programs from a list of ui descriptions. we then use a compiler and vision-language model @xcite to aggressively score, filter, and dededuplicate the output samples to create a refined higher quality dataset. by finetuning on the subset of high-scoring outputs, an improved llm learns to generate ui code that i) successfully compiles and ii) is relevant to the input description. during subsequent iterations, the improved llm generates higher-quality datasets, which results in further performance gains. we call our resulting model uicoder, because we originally started with the open source llm starcoder @xcite . in this paper, we use the latest instruction-tuned version of starcoder, at the time of writing, called starchat-beta, as our base model @xcite . we applied five iterations of our method, resulting in nearly one million generated swiftui programs, and trained three model variations for text-to-ui code generation. in a series of experiments that measured automated metrics and human preference, we show that our models significantly outperform other downloadable baselines and approach the performance of much larger proprietary llms. notably, these results are particularly impressive given our models originate from starcoder, and swift code repositories were accidentally omitted from the training of this model @xcite . the finetuning dataset used to create starchat-beta from starcoder contains just one swift example out of ten thousand total examples. to summarize, the contributions of our work are as follows: we release the weights of our models and a synthetically generated dataset that can be used to train other llms for ui code generation. ## related work llms are typically trained to respond to commands (i.e., instruction-tuning) or perform other types of specific tasks through multiple stages @xcite : i) a base model is pre-trained on unstructured data, ii) the model is finetuned a dataset of human-authored responses to instructions, and iii) the model is further "aligned" through human feedback or ratings of output. recently, this variations of this workflow has been successfully employed to train llms @xcite . for tasks that require expert knowledge such as programming, an especially time and costintensive part of this process is the creation of a human-labeled supervised tuning dataset and collection of human ratings. some efforts have led to the creation of open-source finetuning datasets via volunteer-based labeling @xcite , but the datasets are comparatively limited in size and do not contain many programming-related examples. other approaches have focused on creating these datasets through more automated methods, including self-curation of crawled web data @xcite , and mining language-code examples from code repository commit messages @xcite . instead of using human programmers to write thousands of programs @xcite , another popular approach has been to query the output of strong proprietary models to generate synthetic data, which is then used to "distill" a new model whose weights are known @xcite . while the weights of distilled models are available for download, they have usage restrictions (e.g., non-commercial use, for research purposes only) due to conflicts with the terms-of-service of proprietary models used to train them and are limited to the performance of their "teacher" model @xcite . in our experiments, we compare models trained with all of these techniques as baselines. another class of approaches most closely related to ours uses automated sources of feedback for model improvement. reinforcement learning from ai feedback (rlhaif) is a technique that uses another llm to rate generated output based on a set of natural language guidelines, referred to as a "constitution" @xcite . rest @xcite , an algorithm that was published during the course of our project, is most similar to our work in that it also uses a filter-then-train strategy. however, we focus specifically on scoring methods relevant to ui code generation and show that developer tools (e.g., a code compiler) and visionlanguage models can be successfully employed as feedback sources. from the perspective of ui generation, there are several prior approaches related to ours. sketchplore @xcite and scout @xcite were ui prototyping/design tools that integrated a layout optimizer to generate design suggestions based on a manually-defined objective function. neural networks have also been used to complete partially complete layouts @xcite and generate layouts "from scratch" @xcite or other conditional input @xcite . in contrast to this line of work, we focus on generating compilable ui code rather than pure layouts, which results in an interactive and a more implementation-friendly format. we detail the training procedure used to train llms to generate swiftui code for a ui given its natural language description. swiftui is a toolkit for the swift language that allows cross-platform uis (desktop, tablet, mobile, and watch) to be composed through a domain specific language (dsl). generating swiftui using llms is difficult, due to challenges associated with code generation in general @xcite , and especially poor representation of swiftui programs in publicly available training data and evaluation benchmarks. ## training procedure to improve the generation capabilities of llms, we apply a training procedure based off of previous work @xcite that involves three high-level stages. our approach is novel in that it uses automated feedback from code compilers and visual-language models in place of human annotations. figure 1 shows the overall overview of our training approach that involves i) training (or using a pre-existing) base model, ii) using supervised finetuning, then iii) preference alignment techniques to further improve performance. hyperparameters were set primarily by manual experimentation, and values for various algorithms are given in the appendix. ## training infrastructure we provide details about the infrastructure used to support model training. our setup i) generates large volumes of swiftui code, ii) renders the swiftui code to screenshots, and iii) scores, filters, then dededuplicates the synthetic dataset before finetuning our model. the servers are connected to a cloud storage provider that allows them to share files between each other. code generator. the code generator is used to generate a large number of swiftui programs using our model. we used a mix of v100 and a100 gpus, which was assigned by our organization's internal infrastructure. 1 the code generator repeatedly sampled a description of a ui from our training datasets using uniform random sampling and constructed a prompt using our model's prompt template. a swiftui program was then generated by sampling our model until a pre-defined stop token was reached. under some training configurations, multiple outputs were generated from the same description. generated swiftui programs were periodically uploaded to network storage. ui renderer. the ui renderer was used to convert swiftui programs to rendered screenshots. the ui renderer ran macos with xcode and the ios simulator that repeatedly i) downloads queued swiftui programs, ii) renders them to screenshots, then iii) uploads the results to the network storage. the image asset renderers ran text-to-image models that generated image assets referenced by the swiftui code. we found that including generated image assets in ui screenshots was more effective than inserting placeholders (e.g., grey rectangles in place of images) when computing clip score. before program compilation, we applied automated program repairs (aprs) to try to automatically fix common errors. we focused on lightweight, manually-defined heuristic repairs (as opposed to computationally-expensive llm-based repair methods) that used regular expressions and line numbers parsed from compiler output to match and replace errors in the source code. training server. the model trainer downloaded and collated the i) input descriptions, ii) generated swiftui programs, and iii) the rendered ui screenshots. a text-image matching model @xcite was used to score each rendered ui screenshot by computing the cosine similarities between i) the embedded input description, and ii) the embedded reference screenshot. depending on the training stage, different algorithms for dataset generation and model training were applied. all models were trained using lora method @xcite to improve training speed and efficiency. ## experiments we conducted experiments to i) measure the performance of our model over time, ii) measure the impact of our data on different llms, and iii) compare against the performance of other baselines. ## conclusion in this paper, we introduced a novel method that uses automated tools such as a code compiler and a pre-trained vision-language model to finetune llms to generate ui code from user-provided textual descriptions. unlike other instructionfollowing llms that are trained on expensive human feedback or output from a stronger proprietary model, our technique trains llms entirely using high-quality examples mined from self-generated data. we applied five iterations of our algorithm to an existing open-source llm, resulting in nearly one million generated swiftui programs, which were then filtered and refined to train the uicoder model. in a series of experiments that measured both automated metrics and human preferences, uicoder and other models trained with our method outperformed all other downloadable baselines by a large margin and approached the performance of larger proprietary models.