Code Generation from Natural Language with Less Prior Knowledge and More Monolingual Data (2021)

Abstract: Training datasets for semantic parsing are typically small due to the higher expertise required for annotation than most other NLP tasks. As a result, models for this application usually need additional prior knowledge to be built into the architecture or algorithm. The increased dependency on human experts hinders automation and raises the development and maintenance costs in practice. This work investigates whether a generic transformer-based seq2seq model can achieve competitive performance with minimal code-generation-specific inductive bias design. By exploiting a relatively sizeable monolingual corpus of the target programming language, which is cheap to mine from the web, we achieved 81.03% exact match accuracy on Django and 32.57 BLEU score on CoNaLa. Both are SOTA to the best of our knowledge. This positive evidence highlights a potentially easier path toward building accurate semantic parsers in practice.

## introduction for a machine to act upon users' natural language inputs, a model needs to convert the natural language utterances to machine-understandable meaning representation, i.e. semantic parsing (sp). the output meaning representation is beyond shallow identification of topic, intention, entity or relation, but complex structured objects expressed as logical forms, query language or general-purpose programs. therefore, annotating parallel corpus for semantic parsing requires more costly expertise. sp shares some resemblance with machine translation (mt). however, sp datasets are typically smaller, with only a few thousand to at most tens of thousands of examples, even smaller than most low resource mt problems. simultaneously, because the predicted outputs generally need to be exactly correct to execute and produce the right answer, the accuracy requirement is generally higher than mt. as a result, inductive bias design in architecture and algorithm has been prevalent in the sp literature @xcite @xcite @xcite @xcite . while their progress is remarkable, excessive task-specific expert design makes the models complicated, hard to transfer to new domains, and challenging to deploy in real-world applications. in this work, we look at the opposite end of the spectrum and try to answer the following question: with little inductive bias in the model, and no additional labelled data, is it still possible to achieve competitive performance? this is an important question, as the answer could point to a much shorter road to practical sp without breaking the bank. this paper shows that the answer is encouragingly affirmative. by exploiting a relatively large monolingual corpus of the programming language, a transformer-based seq2seq model @xcite with little sp specific prior could potentially attain results superior to or competitive with the state-of-the-art models specially designed for semantic parsing. our contributions are three-fold: ## previous work on semantic parsing different sources of prior knowledge about the sp problem structure could be exploited. input structure: @xcite adapts the transformer relative position encoding @xcite to express relations among the database schema elements as well as with the input text spans. @xcite proposed a spanbased neural parser with compositional inductive bias built-in. herzig and berant (2020) also leverages a cky-style @xcite @xcite inference to link input features to output codes. output structure: the implicit tree or graph-like structures in the programs can also be exploited. dong and lapata (2016) proposed parent-feeding lstm following the tree structure. @xcite proposed a coarse-to-fine decoding approach. @xcite crafted an intermediate meaning representation to bridge the large gap between input utterance and the output sql queries. @xcite proposed tranx, a more general-purpose transition-based system, to ensure grammaticality of predictions. using tranx, the neural model predicts the linear sequence of ast-tree constructing actions instead of the program tokens. however, a human expert needs to craft the grammar, and the design quality impacts the learning and generalization for the neural nets. sequential models with less sp specific priors have been investigated @xcite @xcite , however, they generally fell short in accuracy comparing to the best of structure-exploiting ones listed above. the most closely related to ours is the work by xu et al. (2020) for incorporating external knowledge from extra datasets, which used a noisy parallel dataset from stackoverflow to pre-train the sp and fine-tuned it on the primary dataset. their approach's main limitation is still the need for (noisy) parallel data, albeit cheaper than the primary labelled set. nonetheless, as we shall see in the experiment section later, our approach achieves better results when using the same amount of data mined from the same source despite ignoring the source sentence. ## background and methodology bert @xcite class of pre-trained models can make up for the lack of inductive bias on the input side to some degree. on the output side, we hope to learn the necessary prior knowledge about the target meaning representation from unlabelled monolingual data. using monolingual data to improve seq2seq models is not new and has been extensively studied in mt before. notable methods include fusion @xcite @xcite , backtranslation (bt) @xcite @xcite , @xcite @xcite , and bt with copied monolingual data @xcite . however, due to more structured outputs, less training data, and different evaluation metrics of exact match correctness instead of bleu, it is unclear if these lessons transfer from mt to sp. so sp-specific investigation is needed. ## experiments for our primary experiments we considered two python datasets namely django and conala. the former is based on django web framework and the latter is annotated code snippets from stackoverflow answers. additionally, we experiment on the sql version of geoquery and atis from finegan-dollak et al. (2018) (with query split), wikisql @xcite , and magic (java) @xcite . python monolingual corpora: conala comes with 600k mined questions from stackoverflow. we ignored the noisy source intents/sentences and just use the python snippets. to be comparable with xu et al. (2020), we also select a corresponding 100k subset version for comparison. see appendix a for details on the sql and java monolingual corpora. ## conclusion this work has shown the possibility to achieve a competitive or even sota performance on semantic parsing with little or no inductive bias design. besides the usual large-scale pre-trained encoders, the key is to exploit relatively large monolingual corpora of the meaning representation. the modified copied monolingual data approach from machine translation literature works well in this extremely low-resource setting. our results point to a promising alternative direction for future progress.