Co-evolving data-driven and NLU -driven Synthesizers for Generating Code in Domain Growth and Data Scarcity (2023)

Abstract: Natural language programming automatically generates code based on a user’s text query. Recent solutions are either data-driven or natural language understanding (NLU)-driven. However, the data-driven synthesizer requires a large number of query-code pairs for training, which hinders its application to low-resource programming languages with growing domains whose functionality and grammar can be actively updated. NLU-driven synthesizers solve this problem, but their code generation is slow and their performance rapidly saturates in the presence of ever-increasing data. In this paper, we propose a circular training framework, Colead, which co-evolves both the data-driven synthesizer and the NLU-driven synthesizer to achieve high-quality code generation in the presence of data scarcity and domain growth. The NLU-driven synthesizer generates query-code pairs to update the data-driven synthesizer, which shares a part of its updated model to improve the NLU-driven synthesizers, enabling the co-evolution of both. Experiments show that Colead gives better results than the baselines in the presence of domain growth and data scarcity, and Colead consistently improves the performance of both data-driven and NLU-driven synthesizers over the co-evolvement.

## introduction natural language (nl) programming aims to automatically generate programming code based on a user's text query @xcite . it has gained increasing research interest in recent years and has a wide range of applications, not only providing an intuitive programming interface that democratizes artificial intelligence to common users @xcite but also reducing the time and labor cost of turning ideas into code implementations @xcite . typical nl programming approaches can be categorized as rule-driven or data-driven. a ruledriven synthesizer generates code through predefined domain-specific rules, which requires expert knowledge. it made progress in early nl programming research @xcite but gradually lost its appeal due to technical difficulties in adapting to different programming languages. in parallel, data-driven synthesizers based on deep learning techniques have dominated recent studies @xcite @xcite . they generate code through a neural network model. training of the models requires large amounts of parallel data @xcite , where each text prompt is paired with a corresponding piece of code. data-driven synthesizers tend to outperform rule-driven synthesizers in many general-purpose languages (gpls), like python, c++, etc., which have massive parallel data available. however, there are plenty of domain-specific languages (dsls) that are low-resource. some dsls are dedicated to a specific application domain and has few usages and scarce parallel data. the problem of data scarcity hinders the application of datadriven synthesizers. another challenging problem is the domain growth of programming languages, which are constantly updated in terms of grammar and functionality. figure 1 shows an example of a data-driven model (codet5) that cannot generate correct code after an api update. to adapt to these updates, data-driven synthesizers require frequent re-training, and it would be labor-intensive and time-consuming to collect new data for the update. natural language understanding (nlu)-driven synthesizers have recently been proposed as a compromise between rule-driven and data-driven synthesizers @xcite @xcite . it circumvents the huge need for parallel data in training by utilizing input user queries and api documentation. whenever the language is updated with a new api and programming grammar, only the api documentation needs to be modified accordingly to generate the new functional code. however, nlu-driven synthesizers have some limitations. to generate code, grammar and dependency graphs need to be built and traversed, which can be time-consuming and a bottleneck to the speed of code generation. as the training data increases, nlu-driven synthesizers quickly reach performance saturation and become inferior to data-driven synthesizers. in this paper, we propose a co-evolving framework, colead (for "co-learning rule and data from documention") that combines nlu-driven and data-driven synthesizers to exploit their complementary strengths and enable code generation in the presence of data scarcity and domain growth @xcite . colead consists of three key components: a retriever, an nlu-driven synthesizer, and a data-driven synthesizer. the retriever fetches information most relevant to the user's text query from the api documentation. given the information, the nlu-driven synthesizer constructs a grammar graph and a dependency graph to generate code. the generated code is paired with the user query to train the data-driven synthesizer, which in turn shares its updated encoder with the nlu-driven synthesizer, leading to the co-evolvement of both. we summarize contributions as follows: ## related work large language models pre-trained on vast amounts of code have achieved significant progress in recent years @xcite . these models can be classified as encoder-only, decoderonly, or encoder-decoder. an encoder-only model predicts masked code fragments based on their surroundings. it converts code into effective vector representations and facilitates a myriad of downstream tasks, such as code summarization (ahmad et al., 2020), code classification (gilda, 2017), and code clone detection @xcite . the representative models include cubert @xcite , @xcite , and @xcite . by contrast, a decoder-only model, such as codegpt @xcite , codegen @xcite , cert @xcite , predicts the next token given the previous tokens in an auto-regressive manner. however, these models are not a perfect fit for natural language programming tasks. an encoder-decoder model first uses an encoder to encode the input sequence and then decodes it with a decoder into an output sequence conditioned on the input sequence. codet5 @xcite , @xcite , polycoder @xcite , and alpha-code @xcite are examples of such models in code. encoder-decoder models perform well on conditional code generation, such as code annotation, natural language programming, etc. note that both decoder-only and encoder-decoder models can be employed directly for code generation. these models all share the drawback of requiring an abundance of data. on the contrary, natural language understandingdriven approaches @xcite ; young et al., 2022; nan et al., 2022) require no training examples. they apply nlp techniques to both natural language (nl) queries and api documentations, extract the key components inside the nl queries and compose the mapped apis into code expressions following the domain grammar. however, nlu-driven methods are not as powerful as datadriven models when there is a lot of data available for training. ## methodology the major challenge of this work is how to efficiently connect the nlu-driven and data-driven synthesizers so that the joined framework possesses the ability to alleviate data scarcity and domain growth problems while ensuring performance limits and inference speed. the colead framework is proposed to address this challenge, as shown in fig 2 . it has three key components: a retriever, an nlu-driven synthesizer, and a data-driven synthesizer. ## experiments experiments are conducted to answer the following research questions: ## discussion we try chatgpt @xcite to accomplish our task and compare it to our approach. to ensure that chatgpt has the same data situation query insert colon before every 1st word ours insert(string(colon), position(before(wordtoken()), integerset(integer(1))), iterationscope(linescope(), bconditionoccurrence(always(), all()))) 0-shot chatgpt insert ":", start all 5-shot chatgpt insert string(colon) position(before(1st()), all()) iterationscope(linescope(), bconditionoccurrence(always(), all())) as our approach, we feed the entire grammar and documentation into chatgpt. we use the prompt "based on the above grammar and documentation, write dsl code to accomplish my query" to instruct chatgpt and prepend 5 query-code pairs in the 5-shot setting. in chatgpt does not require any additional training. it utilizes contextual information to provide inferences, which demonstrates its versatility and adaptability to various tasks. however, if no sample is provided, chatgpt would not perform well. 0-shot chatgpt partially understands the request and outputs plausible code. 5-shot chatgpt generates an almost correct answer, but the grammar is not standardized enough. while correcting it once could easily solve the problem, this step alone incurs significant labor costs, whereas colead is generated strictly according to grammatical rules, and thus has a much smaller probability of grammatical errors. ## conclusion and future work domain growth and data scarcity are two challenges that hinder the application of code generation to dsls. we have shown that our proposed colead framework can effectively mitigate these problems. our framework combines nludriven and data-driven synthesizers, where the nlu-driven synthesizer alleviates the data-hungry issue of the data-driven one and the data-driven synthesizer provides better semantic mapping for the nlu-driven synthesizer to improve code quality. future work should consider code generation by leveraging grammar rules to regularize language models. components in the nlu-driven synthesizer can be further improved, e.g., semantic mapping, nlp engines, etc. more powerful language models, such as gpt3 @xcite , can be leveraged to improve the datadriven synthesizer.