Long-Range Modeling of Source Code Files with e WASH : Extended Window Access by Syntax Hierarchy (2021)

Abstract: Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks, setting high benchmarks for tools in modern software development environments. The finite context window of these neural models means, however, that they will be unable to leverage the entire relevant context of large files and packages for any given task. While there are many efforts to extend the context window, we introduce an architecture-independent approach for leveraging the syntactic hierarchies of source code for incorporating entire file-level context into a fixed-length window. Using concrete syntax trees of each source file we extract syntactic hierarchies and integrate them into context window by selectively removing from view more specific, less relevant scopes for a given task. We evaluate this approach on code generation tasks and joint translation of natural language and source code in Python programming language, achieving a new state-of-the-art in code completion and summarization for Python in the CodeXGLUE benchmark. We also introduce new CodeXGLUE benchmarks for user-experience-motivated tasks: code completion with normalized literals, method body completion/code summarization conditioned on file-level context.

## introduction large transformer models @xcite and the pre-training/fine-tuning paradigm @xcite @xcite have become an essential part of state of the art natural language processing. beyond the domain of natural language, these models and procedures have enabled rapid progress in the software engineering space, including applications in code completion @xcite @xcite @xcite , natural language to code (nl2code), code feature summarization @xcite @xcite @xcite , code search @xcite , unit test generation @xcite and even bug fixing @xcite and detection @xcite . a major difference between transformers and their antecedents like recurrent neural networks (rnn) is their strictly enforced finite context window. whereas an rnn can iteratively consume as many tokens as is required, transformers can only consume up to a finite amount decided at training time. further, it is impractical to simply expand the window as the memory and compute requirements of the attention mechanism scale quadratically with context length. there have been efforts to economically expand the window by modifying the attention mechanism with low-rank queries and keys @xcite , sparse connections @xcite , and more recently approximation by kernel methods @xcite . there have also been meth-ods developed to condition generation on retrieved documents @xcite for knowledgeintensive applications. complimentary to these approaches, and consistent with any sequence model or architecture, we propose a method for extracting the most important features distant from the task at hand, implemented in this case using the syntax tree of source code. a source code document has nested scopes and references to other documents/libraries, and software engineering tasks must leverage knowledge at all of these scales. even single source code files often exceed the length of the context window, and so it is clear that progress in modeling source code requires overcoming this limitation. instead of proposing a novel transformer architecture capable of ingesting more context, we propose a method of compressing the context of source code files using the syntax of the programming language itself. in this paper we introduce ewash: extended window access by syntax hierarchy, which leverages the syntax hierarchy of source code to give our models longer ranged vision by prioritizing higher level scopes and names for source code elements in a file which are not immediately in focus. using ewash assumes that higher level scopes like function signatures summarize the whole method, and viewed in this way ewash could be applied to natural language by extracting key terms in a long document or summarizing features distant from the task at hand. we start by explaining ewash with a motivating example and define its features for three important software engineering tasks. the first is code completion, in which some number of tokens are predicted to extend a partially complete source code file. the second is method completion, wherein a whole method body is predicted from a method signature and docstring. the third is code summarization or docstring generation, wherein a method is mapped to a natural language docstring. we then discuss the python training data and the models employed in this study, which include the auto-regressive gpt-c (svyatkovskiy et al., 2020), an ewash extended version called xgpt-c, python method/docstring prediction model pymt5 @xcite , similarly named xpymt5, and performer @xcite and reformer @xcite baselines. we demonstrate through experiments the value of ewash for the source code domain with state of the art performance for code completion and method completion and code summarization or docstring generation. finally, we study user-experience motivated properties of xgpt-c, and extend the source code benchmark set codexglue @xcite with two new tasks: literal-normalized code completion and method completion. surprisingly, ewash excels at code completion even when the context window is not a limiting factor. ## motivating example in this paper we consider three software engineering tasks: code completion, method completion, and docstring completion or code summarization. code completion is the auto-regressive prediction of one or more tokens conditioned on a provided context. figure 1 shows an incomplete python program implementing a neural network. a developer would like a model which can predict the body of convnet.forward (method completion) by composing the layers defined in convnet.__init__ and globally imported operations. there is a limited context window, illustrated at the left of the figure, and so while the model can (in this fortunate case) see the layer definitions, it is ignorant of the imports and the global logger object. in many cases predicting a whole method is not easy, but asking for a few members or a completed line @xcite ) is still desirable. the bottom of fig. 1 shows the code completion task finishing an assignment and completing a line in a partial implementation of convnet.forward. here, again, crucial information is missing from the input of the model, which will prevent it from being able to reference the imports. further, one can easily imagine a scenario in which more spurious information is fed into the model instead of, for example, the definitions of the neural network layers in the __init__. how can we ensure the model is shown important information for predictions? ## baseline models we consider state-of-the-art transformer models for code completion and code summarization tasks in the codexglue benchmark as our baselines. namely, the generative pre-trained transformer model for code @xcite (gpt-c) and the python method text-to-text transfer transformer model @xcite . we also experiment with two memory-efficient transformers-reformer @xcite and performer (choromanski et al., 2021)-which enables modeling of context lengths in excess of 1024 tokens. ## evaluation 7.1 metrics the metrics we used to evaluate ewash and thus xgpt-c and xpymt5 are consistent with gpt-c and pymt5 and other works in the literature. we report the longest common subsequence rouge-l as we expect in a developer tool scenario that users will want predicted code with the fewest edits. to that end, we also report the edit distance between the truth and hypothesis. in order to compare to other code completion models we report exact-match@n (em) metrics @xcite , which counts the fraction of exactly correct predictions of some length (@n). for method and docstring completion we report bleu-4 and rouge-l metrics, but not exact matches as it is too strict to meaningfully interpret for longer source-target pairs. for method completion we also report the fraction of syntactically correct methods as judged by python 3.8 syntax. ## conclusions inspired by the performance of transformer models, their limited context window size, and the especially long-range nature of source code as documents, we developed extended window access by syntax hierarchy. our hypothesis was that the syntax hierarchy imposed by developers is a real signal of importance in a task context, and that methods, containing most lines of code, are most dependent on the higher-level scopes of their filelevel attributes. our xgpt-c results for code completion supported this hypothesis, and, strikingly, offered most relative benefit for shorter context lengths. we showed with strict exact match metrics that ewash allows a large relative improvement in code completion predictions. finally we show dramatic improvement in method completion and code summarization with xpymt5. ewash can be applied to any programming language and in principal any language with hierarchical syntactic or stylistic structure. for this reason we believe ewash to be a general purpose modeling approach for more optimally using finite context windows on structured documents, and could improve natural language understanding tasks as well. further, any model, even the largest gpt-3 language model (brown et al., 2020) can leverage the ewash feature. accompanying this manuscript we submit 3 new tasks to codexglue to bolster its user-experience motivated metrics: literal-normalized code completion, method-level code completion, and method/docstring completion conditioned on whole-file context.