GAP -Gen: Guided Automatic Python Code Generation (2023)

Abstract: Automatic code generation from natural language descriptions can be highly beneficial during the process of software development. In this work, we propose GAP-Gen, a Guided Automatic Python Code Generation method based on Python syntactic constraints and semantic constraints. We first introduce Python syntactic constraints in the form of Syntax-Flow, which is a simplified version of Abstract Syntax Tree (AST) reducing the size and high complexity of Abstract Syntax Tree but maintaining crucial syntactic information of Python code. In addition to Syntax-Flow, we introduce Variable-Flow which abstracts variable and function names consistently through out the code. In our work, rather than pretraining, we focus on modifying the finetuning process which reduces computational requirements but retains high generation performance on automatic Python code generation task. GAP-Gen fine-tunes the transformer based language models T5 and CodeT5 using the Code-to-Docstring datasets CodeSearchNet, CodeSearchNet AdvTest and Code-Docstring Corpus from EdinburghNLP. Our experiments show that GAP-Gen achieves better results on automatic Python code generation task than previous works

## introduction with billions of people relying on software for their everyday work and life, developers face an ongoing challenge to create programs efficiently. one potential solution to this challenge is to use human descriptions to generate the corresponding source code. by using this approach, developers can write software specifications in natural language, which are then translated into code through code generation mechanism. early attempts to tackle this problem were rule-based, identifying syntactic patterns in text and using handcrafted rules to map the patterns to code. methods used to recognize syntactic structure include regular patterns @xcite @xcite and parse trees produced using context-free grammars @xcite @xcite . several previous approaches convert a sentence into a formal statement by mapping verbs to functions in the formal language, and mapping the objects of the verb in the sentence to function arguments in the formal language @xcite @xcite . for example, the sentence, "add r1 to r2" might be mapped to add(r1, r2) in a procedural language. the problem of finding the objects of the verb to use as function arguments is simple if the sentence structure is strictly limited. several approaches use regular expressions @xcite or context-free grammars @xcite to identify the objects in the sentence. more recent approaches are data-driven and leverage machine learning methods, e.g., @xcite uses a naive bayesian classifier to map english words to a domain-specific language, and @xcite learns production rules for a semantic parser. @xcite uses a long-short term memory (lstm) recurrent neural network (rnn) architecture to implement their neural machine translation approach . work presented in @xcite introduces the latent predictor network (lpn) architecture which treats code generation as a sequence-to-sequence modeling problem. @xcite builds upon this approach by leveraging the grammar model of the target language as prior knowledge. research presented in @xcite @xcite introduced transformer-based language model pre-training methods to map the natural language semantic with the code. although these works have relatively good performance on source code generation task, they require high computational resources, which are difficult to acquire. they also usually consider the code as a sequence of tokens @xcite @xcite and ignore either the source code's syntactic-level or semanticlevel information, which could improve the language models' code understanding capability, during their pre-training process. in this work, we present gap-gen, a method to improve automatic python source code generation from natural language description. our gap-gen is fine-tuning of the pre-trained t5-english @xcite and codet5 @xcite language models that employ syntax-flow and variable-flow as guidance and has shown on being able to understand the relationship between natural language description and python code from syntactic and semantic level of the python code. our gap-gen training pipeline is composed of two phases. as shown in figure 1 , phase i, finetunes our pre-trained language model for the purpose of generating python code's syntactic constraints and semantic-level structure, the syntax-flow and the variable-flow. phase ii, fine-tunes a separate language model by encoding natural language description of the code, the generated code syntactic constrains (syntax-flow) and abstracted variable names (variable-flow) from phase i to generate python code. by doing so, language models fine-tuned with gap-gen training pipeline are able to surpass many previous works' performances, which rely on the pre-training process of language models without considering code's syn-tactic and semantic information. our main contributions are: transformer-based language models that utilize attention mechanisms have been dominating nlp benchmarks @xcite . the novel attention-based message passing techniques plus multi-task pre-training @xcite have been through extensive studies. this leads to a deeper understanding of the representational power of transformer-based models @xcite @xcite . at the same time, transformer-based autoregressive language models consisting of encoder/decoder demonstrate stellar performances on many nlp generative tasks @xcite @xcite . these tasks include but not limited to story generation @xcite , dialogue @xcite , summarization @xcite , entity @xcite , question answering @xcite , and so on. similar advances have also been made in programming language relevant tasks. programming language generation tasks, although not considered as natural language generation tasks, have been demonstrated to have great results when they are modeled similarly as natural language generation tasks. @xcite pre-trains on mask language modeling (mlm) and replaced-token detection for code understanding tas. in @xcite , the authors develop a code completion transformer-based model by jointly predicting the probability and type of the next token. for the task of code summarization, transformer-based models outperform the other neural approaches @xcite ; @xcite use gpt and unilm respectively for code completion. more related to our work, @xcite explore pre-training methodologies for learning better structural and syntactical information for automatic code generation. moreover, @xcite incorporates variable-flows and identifier information into their pre-training process for better code generation performance. guided text generative models. generative modeling is powerful but often falls short in many conditions. the behavior of auto-regressive language models cannot be explicitly controlled, and was shown to be very easy to degenerate @xcite @xcite . this is also the case for code generation. this prompts researchers to combat this issue by looking at either the training time or the decoding time. work in @xcite constrains the sample space to top-k tokens in the softmax logistics to avoid introducing highly unlikely tokens. @xcite instead restricts the sampling space to the smallest set of space above some probability mass. using simple decoding variants is lightweight to implement, but does not change the predicted likelihood of each token. @xcite argues that the likelihood objectives is at fault, and proposes unlikelihood training objective, which forces lower probabilities on unlikely generations. furthermore, practitioners have also injected priors or structural information into the language model for better generation. @xcite utilizes policy learning to control model behaviors. however, this approach suffers from high variance @xcite . recently work in story generation @xcite @xcite uses a plotline/storyline as an intermediate state for generation. this alleviates the language modeling tasks and sets up the model to better learn the structure of the stories. being motivated by story generation, our work injects syntactic and semantic structural information in a setup that is similar to this line of works. for the code generation task, we utilize our proposed syntax-flow and variable-flow as the intermediate state to help language model better understand code's syntactic and semantic structure information and improve its performance. ## method in this section, we describe our method by introducing syntax-flow and variable-flow. then we present the generation process of syntax-flow and variable-flow. finally, we present the python code generation process guided by syntax-flow and variable-flow. ## experiment in this section, we present our experiment in detail. first, we introduce the datasets we use and our data processing approach in our experiment. then, we present our experimental setup. finally, we introduce our evaluation metrics in the last subsections. ## conclusions in this work, we demonstrate the effectiveness of injecting python syntactic and semantic information into the code generation tasks. we design and implement two different types of information components: syntax-flow and variable-flow. to incorporate this information, we encode them using separate language models and then feed them along with the docstring input into the final lan-guage model. pre-trained language models finetuned with our proposed pipeline show better performances over state-of-the-art code generation models. for future directions, new strategies for incorporating that information can be explored.