Compilable Neural Code Generation with Compiler Feedback (2022)

Abstract: Automatically generating compilable programs with (or without) natural language descriptions has always been a touchstone problem for computational linguistics and automated software engineering. Existing deep-learning approaches model code generation as text generation, either constrained by grammar structures in decoder, or driven by pre-trained language models on large-scale code corpus (e.g., CodeGPT, PLBART, and CodeT5). However, few of them account for compilability of the generated programs. To improve compilability of the generated programs, this paper proposes COMPCODER, a three-stage pipeline utilizing compiler feedback for compilable code generation, including language model fine-tuning, compilability reinforcement, and compilability discrimination. Comprehensive experiments on two code generation tasks demonstrate the effectiveness of our proposed approach, improving the success rate of compilation from 44.18 to 89.18 in code completion on average and from 70.3 to 96.2 in text-to-code generation, respectively, when comparing with the state-of-the-art CodeGPT.

## introduction automated code generation (or program synthesis) has attracted much attention over the past few years @xcite , because of its potential to improve the productivity of developers, as well as to speed up the software development @xcite . in the life cycle of software development, different types of code generation tasks are desired, including code completion @xcite , text-to-code generation @xcite , program translation @xcite , and program repair @xcite . recently, much effort has been made to advance the development of code generation @xcite , using different logical forms of code, such as the abstract syntax tree (ast) @xcite @xcite , sketch @xcite and graph @xcite . benefiting from the strong power of pre-training techniques @xcite in natural language processing, several attempts have been made towards pretraining a language model on large-scale code corpus for code generation, such as codegpt @xcite , @xcite , and codet5 @xcite . however, to the best of our knowledge, most deep-learning approaches for code generation are still difficult to guarantee the compilability of the generated code, resulting in non-compilable code. for example, @xcite found that up to 67%-97% of patches generated by the most advanced deep-learning-based models are non-compilable. we think this is because they generally do not directly optimize the compilability for code generation. the generation of non-compilable code will waste the time of programmers, as well as seriously reduce the trust and satisfaction of developers with the model. to improve the compilability of the generated code, some works attempt to repair the synthesized program which fails to compile @xcite @xcite . recently, @xcite attempt to directly generate compilable code using an energy model with compilability constraints. this paper focuses on the task of compilable neural code generation. different from previous works, we use compilability signals in two ways and design a novel method to jointly train the discriminator and generator for compilable code generation. concretely, we propose compcoder, a novel three-stage pipeline utilizing compiler feedback for compilable code generation, including language model fine-tuning, compilability reinforcement, and compilability discrimination. figure 1 shows an example of python code completion by compcoder, which utilizes the compiler feedback in two ways. in figure 1 (b), we use the compiler feedback to optimize the generator. in figure 1(c), we use the discriminator to check if the results generated by the generator can be successfully compiled. the joint training of the generator and discriminator significantly improves the compilability of the generated code. overall, the key contributions of this paper are as follows: ## preliminary in this section, we set out notations for task formulation, as well as some preliminaries of compiler feedback. let @xmath0, which can be a piece of partial code, natural-language description, or buggy program. let @xmath1. formally, the problem of code generation can be formulated as learning a mapping f between the input space and target code space, i.e. f : @xmath2. in this paper, we investigate two specific code generation tasks, code completion and text-to-code generation, conditioned on different inputs. ## compcoder figure 2 shows the overall architecture of comp-coder on the code completion task, which covers three stages, i.e., language model fine-tuning (stage 1), compilability reinforcement (stage 2) and compilability discrimination (stage 3). in the following generator training procedure (a) lm fine-tuning generator generator (b) compilability reinforcement generator policy training discriminating loss active model π reference model ρ (c) compilability discrimination --(a) (b) (c) generator discriminator joint learning p(.|t,s) ✖/✔ compiler feedback ... generating candidates compiler generating loss + feedback reward kl-divergence code snippet 1 def func(x,y): 2 @xmath3,y): 2 @xmath4,y): 2 @xmath5,y): 2 @xmath6,y): 2 @xmath7, we will elaborate on each stage one by one. we alternately perform stages 2 and 3, as described in section 3.4. ## conclusion and future work in this paper, we use the compilability signals in two ways and design a novel method to jointly train the generator and discriminator for compilable code generation, called compcoder. comprehensive experiments on two code generation tasks demonstrate the effectiveness of compcoder, improving the average compilation rate of state-ofthe-art codegpt from 44.18 to 89.18 in the code completion task and from 70.3 to 96.2 in the textto-code generation task. this work presents our preliminary attempt to generate compilable code. yet, considering the compilation rate is not the whole story as it still cannot guarantee the correctness of generated code. as a future work, we would like to utilize unit tests to evaluate the code correctness towards building more useful code generation models.