P y MT 5: multi-mode translation of natural language and Python code with transformers (2020)

Abstract: Simultaneously modeling source code and natural language has many exciting applications in automated software development and understanding. Pursuant to achieving such technology, we introduce PyMT5, the Python method text-to-text transfer transformer, which is trained to translate between all pairs of Python method feature combinations: a single model that can both predict whole methods from natural language documentation strings (docstrings) and summarize code into docstrings of any common style. We present an analysis and modeling effort of a large-scale parallel corpus of 26 million Python methods and 7.7 million method-docstring pairs, demonstrating that for docstring and method generation, PyMT5 outperforms similarly-sized auto-regressive language models (GPT2) which were English pre-trained or randomly initialized. On the CodeSearchNet test set, our best model predicts 92.1% syntactically correct method bodies, achieved a BLEU score of 8.59 for method generation and 16.3 for docstring generation (summarization), and achieved a ROUGE-L F-score of 24.8 for method generation and 36.7 for docstring generation.

## introduction software is a keystone of modern society, touching billions of people through services and devices daily. writing and documenting the source code of this software are challenging and labor-intensive tasks; software developers need to repeatedly refer to online documentation resources in order to understand existing code bases to make progress. developer productivity can be improved by the presence of source code documentation and a development environment featuring intelligent, machine-learning-based code completion and analysis tools. recent progress in natural language processing (nlp), especially encoder/decoder-based transformer models @xcite and pre-training @xcite , has led to state-of-the-art performance on language modeling, classification @xcite , translation @xcite , summarization @xcite , grammar correction @xcite , entity recognition, dialogue generation @xcite , and more. along with these quantitative advances have come deeper understanding of the learned hidden representations which power transformers @xcite @xcite . while they are arguably not 'natural,' programming languages are increasingly becoming modeling playgrounds for nlp modeling. since these languages by definition have a grammar, syntax, and known relationships between entities, they offer enticing opportunities for an even deeper probing of nlp models and tasks. beyond theoretical importance, many nlp tasks have practical utility in software development environments: language modeling or generation can be used for code completion @xcite @xcite , translation/summarization to generate documentation or natural language summaries @xcite @xcite or even summarize a set of code changes @xcite , translation and grammar error correction to patch and detect bugs @xcite , and joint embedding of code and natural language for code search @xcite . in this work we focus on jointly modeling both source code (python) and concomitant natural language documentation (docstrings) with transformers, through the study of dual tasks: generating method code bodies from signatures and docstrings, and generating docstrings from signatures and method code bodies. while previous work @xcite has leveraged the grammar of code to extract features like the abstract syntax tree for modeling (treating code and natural language as separate modalities), we follow examples like barone and sennrich (2017) and treat python and its docstrings as fundamentally no different than other 'natural' languages, representing both source code and natural language docstrings as sequences of tokens sharing the same vocabulary. here we present a multi-mode translation method resulting in pymt5, the python method text-to-text transfer transformer (inspired by the text-to-text transfer transformer t5 @xcite ). our single model can both learn code/language generation and understand the relationships between them. the paper is organized as follows: we begin in sec. 2 by presenting examples of the performance of our novel multi-mode pymt5 -the python method text-to-text transfer transformer model-which we trained to translate between all pairs of combinations of method signatures, docstrings, and bodies which do not have the same feature in both the source and target. in sec. 2.1 we describe our training data and the pre-processing steps for source code and natural language we followed, and compared it to existing parallel docstring-method corpora like code-searchnet (csn) @xcite and that presented by @xcite . in sec.2.2 we explain our bartlike (lewis et al., 2019) pre-training scheme, demonstrating a 25× speed-up in training time for docstring generation. next, in sec. 2.3 we analyze and classify python docstrings, enabling style-conditioned docstring generation in pymt5. in sections 3 and 4, we discuss pymt5 results on method generation and docstring generation respectively and compare it to two gpt2 models randomly initialized and pre-trained on english. ## multi-mode training figure 1 shows examples of inputs and outputs of our model pymt5 for 3 example tasks: (top, blue) predicting a body from a method pymt5 figure 2: pymt5 performing docstring generation on an example method, showing the output when the target prefix indicates one line (top, blue) and numpydoc docstring (bottom, red) styles. pymt5 can also be prompted with source code to produce a docstring summary in various styles. figure 2 shows the model prompted with one of the methods generated by pymt5 in fig. 1 (top, blue), in both a 'one line' (top, blue) style and a 'numpydoc' (bottom, red) style. it infers the intent from the signature name and code, and even infers that type of the argument is a list and return type int. it produces the same terse one sentence summary of the function in both cases. in order to teach pymt5 to maximally relate the separate method features (signatures, docstrings, bodies), we trained it to translate between all pairs of feature combinations in which the same feature does not appear in both the source and target. this scheme is also advantageous as our corpus is unbalanced, with only 1/5 methods featuring docstrings, and so the model can learn to leverage all the features whether they are present or not. additionally, it has been shown that code is more 'predictable' than natural language @xcite . if the method and argument names are a dominating signal due to their relatively rigid structure, the model may learn to ignore the content of docstrings. this multi-mode method overcomes that by training the model to generate method bodies from docstrings alone. see the appendix for a more detailed description of the multi-mode training scheme. ## method generation now we turn our attention to method generation: predicting a whole method code body from either a method signature, a natural language docstring, or both. we first discuss a benchmark of this task using a gpt2-medium model (345 million parameters, see the appendix for details), training from scratch and starting with the publicly released openai english pre-trained checkpoint with weights from huggingface @xcite . in all experiments we used an extended gpt2 tokenizerincluding white-space (one tab, two tabs, etc.) tokens-for a total vocabulary size of 50337, and we used beam decoding with a beam width of 5. the third row of tab. 3 shows pymt5 has more than double the bleu score, overall better recall, and significantly better rouge-2 and rouge-l f-scores than our gpt2 baselines. further, 93.6% of the methods generated by pymt5 were syntactically correct python 3.7, whereas only 86% of gpt2 methods were syntactically correct. pymt5 was trained on 16 tesla v100 16gb gpus for 62 epochs, or 5 weeks training time (see the appendix for its hyper-parameters) and the gpt2 baselines were trained on the same hardware for 1 week training time (achieving the same or better validation loss/perplexity as pymt5). the english pre-trained initialization of gpt2 only slightly beats the random initialization of gpt2, which could indicate that the learned biases of english are not particularly beneficial for writing python code; the metrics are almost all within our margin of error. note that barone and sennrich (2017) also modeled methods from docstrings, obtaining a similar bleu score of 10.9 on their own python parallel corpus. the third and fourth rows of tab. 3 show the performance of pymt5 using the publicly available csn python test set, from which we find notably worse results than on our own test set. csn curated their whole set by removing any methods with 'test' in the name and any methods with fewer than 3 lines of code. we calculated the performance of pymt5 only on a subset of our test set curated the same way as csn, observing f-scores for r1, r2, and r-l on our test set of 29.7, 17.2, and 26.1, which is lower than our nominal test set performance of 35.1, 21.5, and 32.2 and closer to the csn performance of 28.4, 13.5, and 24.8. we believe this curating choice explains the differ-ence between our test set and the csn test set. we also conclude that tests and short methods are 'easier' to complete, which is plausible, and bodes well for automatic code completion applications. ## docstring generation we now examine results from the docstring generation task, which for evaluation purposes were conditioned on both signatures and method bodies. as in method generation, we set a gpt2 benchmark with random initialization and pre-trained english initialization as well as the same hyperparameters. we again present the results from the publicly available csn test set. similar to the method generation task, pymt5 performs worse on the csn data than our own, likely for the same reasons we discussed in sec. 3. we also evaluated pymt5 on the barone et al. parallel test set, as shown in the second to last row of tab. 4, and find pymt5 performs notably worse on barone's test set than our own test set, contradicting the hypothesis that our doubling of the method generation bleu score is due to data leakage. pymt5 has a much higher bleu score than that reported by barone et al, perhaps indicating real progress in the code summarization field. docstring generation is similar to code summarization, though the domains are different as docstrings also contain structured annotations of arguments, return values, raised exceptions, and even in-line unit tests (doctest). trans 3 by wang et al. @xcite reports a best rouge-l of 51.27 on the same test set for code summarization, but does not specify which statistic they are reporting, so we cannot make strong conclusions about the performance of pymt5 compared to the state of the art. ## conclusion in this work, we presented a novel multi-mode python method text-to-text transfer transformer model pymt5as well as the largest parallel corpus of python source code and docstrings reported in the literature to date. we have trained pymt5 to translate between all pairs of combinations of method signatures, docstrings, and method bodies which do not have the same feature in both the source and target. further, we introduced control token prefixes for docstring generation to facilitate docstring generation of various styles. focusing on two modeling tasks -predicting python methods from docstrings and summarizing python source code methods into docstrings of various commonly occurring styles -we have compared this new approach to the auto-regressive gpt2 baselines trained on individual docstring or method generation tasks. on the codesearchnet test set pymt5 achieves a bleu score of 8.59 for method generation and 16.3 for docstring generation, and a rouge-l f-score of 24.8 for method generation and 36.7 for docstring generation. we have demonstrated the effectiveness of dynamic masked pre-training, reducing docstring generation training time by 25×. looking forward, we plan to leverage pymt5 for various downstream automated software engineering tasks-including code documentation and method generation from natural language statements-and develop more model evaluation criteria to leverage the unique properties of source codes.