A Transformer-based Approach for Source Code Summarization (2020)

Abstract: Generating a readable summary that describes the functionality of a program is known as source code summarization. In this task, learning code representation by modeling the pairwise relationship between code tokens to capture their long-range dependencies is crucial. To learn code representation for summarization, we explore the Transformer model that uses a self-attention mechanism and has shown to be effective in capturing long-range dependencies. In this work, we show that despite the approach is simple, it outperforms the state-of-the-art techniques by a significant margin. We perform extensive analysis and ablation studies that reveal several important findings, e.g., the absolute encoding of source code tokens’ position hinders, while relative encoding significantly improves the summarization performance. We have made our code publicly available to facilitate future research.

## introduction program comprehension is an indispensable ingredient of software development and maintenance @xcite . a natural language summary of source code facilitates program comprehension by reducing developers' efforts significantly @xcite . source code summarization refers to the task of creating readable summaries that describe the functionality of a program. with the advancement of deep learning and the availability of large-scale data through a vast number of open-source repositories, automatic source code summarizing has drawn attention from researchers. most of the neural approaches generate source code summaries in a sequence-to-sequence fashion. one of the initial works @xcite trained an embedding matrix to represent the individual code tokens and combine them with a re-current neural network (rnn) via an attention mechanism to generate a natural language summary. subsequent works @xcite adopted the traditional rnnbased sequence-to-sequence network @xcite with attention mechanism @xcite on different abstractions of code. the rnn-based sequence models have two limitations in learning source code representations. first, they do not model the non-sequential structure of source code as they process the code tokens sequentially. second, source code can be very long, and thus rnn-based models may fail to capture the long-range dependencies between code tokens. in contrast to the rnn-based models, transformer @xcite , which leverages self-attention mechanism, can capture long-range dependencies. transformers have been shown to perform well on many natural language generation tasks such as machine translation @xcite ), text summarization @xcite , story generation @xcite , etc. to learn the order of tokens in a sequence or to model the relationship between tokens, transformer requires to be injected with positional encodings @xcite @xcite . in this work, we show that, by modeling the pairwise relationship between source code tokens using relative position representation @xcite , we can achieve significant improvements over learning sequence information of code tokens using absolute position representation @xcite . we want to emphasize that our proposed approach is simple but effective as it outperforms the fancy and sophisticated state-of-the-art source code summarization techniques by a significant margin. we perform experiments on two wellstudied datasets collected from github, and the results endorse the effectiveness of our approach over the state-of-the-art solutions. in addition, we provide a detailed ablation study to quantify the effect of several design choices in the transformer to deliver a strong baseline for future research. ## proposed approach we propose to use transformer @xcite to generate a natural language summary given a piece of source code. both the code and summary is a sequence of tokens that are represented by a sequence of vectors, @xmath0, . . . , x n ) where x @xmath1. in this section, we briefly describe the transformer architecture ( § 2.1) and how to model the order of source code tokens or their pairwise relationship ( § 2.2) in transformer. ## related work most of the neural source code summarization approaches frame the problem as a sequence generation task and use recurrent encoder-decoder networks with attention mechanisms as the fundamental building blocks @xcite @xcite . different from these works, @xcite proposed a convolutional attention model to summarize the source codes into short, name-like summaries. recent works in code summarization utilize structural information of a program in the form of abstract syntax tree (ast) that can be encoded using tree structure encoders such as tree-lstm @xcite , tree-transformer @xcite . in contrast, @xcite proposed a structure based traversal (sbt) method to flatten the ast into a sequence and showed improvement over the ast based methods. later, @xcite used the sbt method and de-coupled the code structure from the code tokens to learn better structure representation. among other noteworthy works, api usage information @xcite , reinforcement learning @xcite , dual learning @xcite , retrieval-based techniques @xcite are leveraged to further enhance the code summarization models. we can enhance a transformer with previously proposed techniques; however, in this work, we limit ourselves to study different design choices for a transformer without breaking its' core architectural design philosophy. ## conclusion this paper empirically investigates the advantage of using the transformer model for the source code summarization task. we demonstrate that the transformer with relative position representations and copy attention outperforms state-of-the-art approaches by a large margin. in our future work, we want to study the effective incorporation of code structure into the transformer and apply the techniques in other software engineering sequence generation tasks (e.g., commit message generation for source code changes). we can see from