Reading S tack O verflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation (2021)

Abstract: Answering a programming question with only its title is difficult as salient contextual information is left out. To address this, we present a corpus of over 40,000 StackOverflow question texts to be used in conjunction with the corresponding intents from the CoNaLa dataset (Yin et al., 2018). Using both the intent and the question body, we use BART to establish a baseline BLEU score of 34.35 for this new task. We then find further improvements of 2.8% by combining the mined CoNaLa data with the labeled data to achieve a 35.32 BLEU score. We then evaluate the prior state-of-the-art CoNaLa models with this additional data. We find that our proposed method of using the body and mined data beats that of the previous state-of-the-art by a 71.96% BLEU score. Finally, we perform ablations that prove that BART is an unsupervised multimodal learner and examine its extractive behavior.

## introduction the goal of semantic parsing is to translate a natural language(nl) utterance to its logical components. there is a large body of research on applying semantic parsing for source code generation in a multitude of domain specific languages such as lambda calculus and sql @xcite @xcite @xcite @xcite @xcite . however, the task of translating an nl utterance to a general-purpose programming language has proven to be more challenging. a significant issue contributing to this is the difficulty in acquiring quality data due to the necessary domain knowledge needed in the annotation process. despite this, the past few years have seen a large number of datasets released for different text-to-figure 1 : overview of our approach. from the combined annotated + mined set, we concatenate the intent and question body for inputs to bart @xcite and use beam search for generation. transformers @xcite have demonstrated that they can both be few-shot @xcite and unsupervised multitask @xcite learners. they have been successfully applied to programming language tasks. codebert achieved strong performance on the codesearch-net task through pretraining on bimodal nl comment and code pairs @xcite , while @xcite used abstract syntax trees(ast) and transformers to achieve state of the art performance on the hearthstone benchmark @xcite . @xcite proposed the deobfuscation pretraining task to incorporate structural features of code into transformer models without the use of asts. more recently, @xcite explored the capabilities of large pretrained language models to be few-shot semantic parsers. yet open-domain programming question answering on sites such as stackoverflow(so) foot_1 has remained an elusive goal. @xcite created an annotated dataset with the site in which the intent and answer snippet pairs were automatically mined from the question. they then had crowd workers rewrite the intents to reflect the corresponding code better. currently, state-of-the-art was achieved by pretraining an lstm model on resampled api and mined data @xcite . subsequent work conducted an empirical study on the effectiveness of using a code generation model in an ide plugin and find that developers largely had favorable opinions of their experience @xcite . an inherent issue with the approach of xu et al. ( 2020 ), more fundamentally the dataset and parameters of the task, is that the intent can only contain a limited amount of information. consider the question from figure 2 in which a valid python snippet could be a[:, (np.newaxis)]. arriving at this answer from the intent "add a new axis to array a" requires not only the disambiguation of data types for variable a, but also the use of multiple distinct library-specific concepts. further, this must be accomplished while maintaining syntactically correct code and proper order of arguments. however, neither the original title nor the rewritten intent contains the necessary information to accomplish this task. although the previous state-of-the-art-model by @xcite uses abstract syntax trees (ast) to guarantee syntactically valid python code, it incorrectly generates a[(-1),:]=a. one potential remedy would be to increase the amount of training data, but as discussed previously, getting high-quality annotated code generation data is especially difficult. motivated by the limitations to the amount of information a given intent can contain and the substantial difficulty involved with gathering more labeled data, we utilize the multimodal text from the question bodies provided by the stackexchange api foot_2 . we take advantage of the strong performances of transformer models to beat the previous state-of-the-art by 3.06 bleu. we ensure a fair comparison by training the models from prior works with the extra data to adequately evaluate our proposed method. when all models are trained with the extra data, using bart beats the previous state of the art by 15.12 bleu. our main contributions are the following: ## methodology as detailed in figure 1 , our overarching approach is to: (1) gather textual bodies from so for both the annotated and mined examples in the conala corpus, (2) use the concatenated intents and question bodies as inputs for a large pretrained language model, and (3) use beam search to generate the answer code snippet. ## results the 71.96% increase is likely because tranx models were never intended to perform well with very noisy data, as evidenced by the 36% dropoff in corpus bleu when adding the body to @xcite . in choosing bart, we intentionally picked a transformer model designed for denoising @xcite . further testing is likely needed to determine if our approach is heavily dependent on the underlying transformer, but that is beyond the scope of this paper. ## conclusion we expand the conala dataset by adding the textual question bodies from the stackexchange api and achieve state-of-the-art performance with a simple bart model. further, we demonstrate that, for this task, bart performs best when code blocks, inline code, and nl are all present. we then examine the impact of the question body on syntax errors and bart's cheating through multimodal understanding. finally, we examine examples that highlight the issues with both stackoverflow data and code evaluation in general. future work should focus on extracting desired inputs and outputs for a given intent. further, additional efforts put into creating corpora of executable code are likely to improve not only generation but evaluation. both will also protect datasets from deprecated functions and abandoned libraries.