Incorporating External Knowledge through Pre-training for Natural Language to Code Generation (2020)

Abstract: Open-domain code generation aims to generate code in a general-purpose programming language (such as Python) from natural language (NL) intents. Motivated by the intuition that developers usually retrieve resources on the web when writing code, we explore the effectiveness of incorporating two varieties of external knowledge into NL-to-code generation: automatically mined NL-code pairs from the online programming QA forum StackOverflow and programming language API documentation. Our evaluations show that combining the two sources with data augmentation and retrieval-based data re-sampling improves the current state-of-the-art by up to 2.2% absolute BLEU score on the code generation testbed CoNaLa. The code and resources are available at https://github.com/neulab/external-knowledge-codegen .

## introduction semantic parsing, the task of generating machine executable meaning representations from natural language (nl) intents, has generally focused on limited domains @xcite , or domain-specific languages with a limited set of operators @xcite @xcite @xcite @xcite . however, recently there has been a move towards applying semantic parsing to automatically generating source code in general-purpose programming languages @xcite @xcite @xcite . prior work in this area (xiao et al., 2016; ling et al., 2016; rabinovich et al., 2017; yin and neubig, 2017, 2018; dong and lapata, 2018; suhr et al., * the first two authors contributed equally. 2018; @xcite used a variety of models, especially neural architectures, to achieve good performance. however, open-domain code generation for general-purpose languages like python is challenging. for example, given the intent to choose a random file from the directory contents of the c drive, 'c:\\', one would expect the python code snippet random.choice(os.listdir('c:\\')), that realizes the given intent. this would involve not just generating syntactically correct code, but also using (and potentially combining) calls to apis and libraries that implement some of the desired functionality. as we show in § 3, current code generation models still have difficulty generating the correct function calls with appropriate argument placement. for example, given the nl intent above, although the state-of-the-art model by @xcite that uses a transition-based method to generate python abstract syntax trees is guaranteed to generate syntactically correct code, it still incorrectly outputs random.savefig(random( compile(open('c:\\'))+100).isoformat()). a known bottleneck to training more accurate code generation models is the limited number of manually annotated training pairs available in existing human-curated datasets, which are insufficient to cover the myriad of ways in which some complex functionality could be implemented in code. however, increasing the size of labeled datasets through additional human annotation is relatively expensive. it is also the case that human developers rarely reference such paired examples of nl and code, and rather take external resources on the web and modify them into the desired form @xcite @xcite . motivated by these facts, we propose to improve the performance of code generation models through a novel training strategy: pretraining the model on data extracted automatically from external knowledge resources such as existing api documentation, before fine-tuning it on a small manually curated dataset ( § 2.1). our approach, outlined in figure 1 , combines pairs of nl intents and code snippets mined automatically from the q&a website stackoverflow ( § 2.2), and api documentation for common software libraries ( § 2.3). 1 while our approach is model-agnostic and generally applicable, we implement it on top of a state-of-the-art syntax-based method for code generation, tranx @xcite , with additional hypothesis reranking @xcite . experiments on the conala benchmark @xcite show that incorporating external knowledge through our proposed methods increases bleu score from 30.1 to 32.3, outperforming the previous state-of-the-art model by up to 2.2% absolute. qualitatively analyzing a sample of code snippets generated by our model reveals that the generated code is more likely to use the correct api calls for desired functionality and to arrange arguments in the right order. ## conclusion and future work we proposed a model-agnostic approach based on data augmentation, retrieval and data re-sampling, to incorporate external knowledge into code generation models, which achieved state-of-the-art results on the conala open-domain code generation task. in the future, evaluation by automatically executing generated code with test cases could be a better way to assess code generation results. it will also likely be useful to generalize our re-sampling procedures to zero-shot scenarios, where a programmer writes a library and documents it, but nobody has used it yet. for example, developers may provide relative estimates of each documented api usages to guide the re-sampling; or we could find nearest neighbors to each api call in terms of semantics and use existing usage statistics as estimates to guide the re-sampling.