Novel Natural Language Summarization of Program Code via Leveraging Multiple Input Representations (2021)

Abstract: The lack of description of a given program code acts as a big hurdle to those developers new to the code base for its understanding. To tackle this problem, previous work on code summarization, the task of automatically generating code description given a piece of code reported that an auxiliary learning model trained to produce API (Application Programming Interface) embeddings showed promising results when applied to a downstream, code summarization model. However, different codes having different summaries can have the same set of API sequences. If we train a model to generate summaries given an API sequence, the model will not be able to learn effectively. Nevertheless, we note that the API sequence can still be useful and has not been actively utilized. This work proposes a novel multi-task approach that simultaneously trains two similar tasks: 1) summarizing a given code (code to summary), and 2) summarizing a given API sequence (API sequence to summary). We propose a novel code-level encoder based on BERT capable of expressing the semantics of code, and obtain representations for every line of code. Our work is the first code summarization work that utilizes a natural language-based contextual pre-trained language model in its encoder. We evaluate our approach using two common datasets (Java and Python) that have been widely used in previous studies. Our experimental results show that our multi-task approach improves over the baselines and achieves the new state-of-the-art.

## introduction developers spend the most time writing code but not much in writing its description. it is reported that much of the developers' code does not have any description @xcite . this has detrimental effects on other developers who will be reading and trying to understand the code base @xcite . to alleviate effort in writing the code description, code summarization, the task of automatically generating code description given a piece of code, has been proposed in the software engineering and ai community @xcite @xcite . previous work on code summarization using an auxiliary model trained to produce api (application programming interface) embeddings showed promising results when applied to a separate code summarization model @xcite . however, different code may assume the same set of api sequence. for example, figure 1a and 1b show two different code snippets having the same api sequence, getarea. despite having the same api sequence, the code summary shown in the comments on top is different: figure 1a is about getting an area given a name, while figure 1b is about doubling an area. thus, training a model to summarize the given code based on its api sequence may induce confusion into the model. nevertheless, we note that the api sequence can still be useful and has not been actively explored. in this work, we further leverage the api sequence for code summarization. specifically, we propose a novel multi-task approach that simultaneously trains two similar tasks: 1) code to summary, and 2) api sequence to summary. our model consists of an encoder-decoder architecture. that is, we propose a novel code-level encoder based on bert @xcite , which has recently shown remarkable improvement in numerous nlp downstream tasks. our encoder is able to express the semantics of code and obtain modeling for every line of code. our work is also the first code summarization work that utilizes a natural languagebased contextual pre-trained language model in its encoder. for multi-task learning, the two different tasks utilize the same set of shared layers that produce contextual embeddings to further train the individual tasks. we evaluate our approach on two popular datasets (java and python) that have been widely used in previous studies. our experimental results show that by learning to identify lines of code, our model is able to learn more effectively. furthermore, our multi-task approach improves over the baselines and achieves the new state-of-the-art performance. in summary, our key contributions include: ## related work most existing approaches that perform code summarization using neural networks define the output task to be a sequence generation @xcite @xcite . these approaches leverage recurrent encoder-decoder models with attention mechanisms. one prior work proposed a new convolutional attention model for code summarization that outputs short name-like summaries @xcite . different from this previous work, our work is a multi-task (code to summary and api sequence to summary) approach that utilizes a contextual pre-trained language model in the area of code summarization. there has been research that leverages source code representation for code summarization. a number of recent work transforms the source code into abstract syntax tree (ast) and encodes it using treetransformer @xcite , tree-lstm @xcite , and graph neural networks @xcite . another prior work has improved the code summarization using the ast by flattening it into a sequence through structure-based traversal @xcite . later work further improved the model by proposing a representation that decouples the code structure from structure-based traversal @xcite . our work leverages a more readable and simplified structure than ast by tokenizing every line of code (e.g., @xcite other existing work leverages various learning techniques such as reinforcement learning @xcite , dual learning @xcite , and retrieval-based techniques @xcite to build the code summarization models. recent work leverages a transformer model to generate natural language summary @xcite ). in contrast, our work uses a pre-trained language model as a code-level encoder. a previous technique uses api usage information that enhances the code summarization model, showing the effectiveness of the knowledge from api sequence @xcite . we noted the exploration of api sequence in code summarization is limited. although multiple different codes can have the same api sequence and the api sequence may not contain the full information of the code, the api sequence can directly provide more structural information of the code, e.g., the data type used, and their sequences. this can be viewed as a friendlier/structured/cleaner intermediate representation. though it may be lossy in the code, such intermediate representation facilitates the model training as we have empirically shown that it is improving the model performance. contrary to @xcite , our work consists of a multi-task approach that summarizes code as well as api sequence. our experiments show that this multi-task approach is useful in learning a better code-to-summary model. there has been work that uses a dual task model @xcite in which one model is trained in step s, and the other model is trained in step s + 1. the result of the model in step s is then used for the model in step s + 1. this cycle keeps on repeating until convergence. our work does not require cycle dependency between two models but simultaneously trains two tasks, which is more efficient. codebert @xcite and pymt5 @xcite present pre-trained models by using a multi-layer bidirectional transformer encoder @xcite and a text-to-text transfer transformer t5 @xcite , respectively. codebert @xcite and pymt5 @xcite both support multiple downstream tasks such as code search @xcite , code generation @xcite , and code summarization @xcite . different from these techniques, our model is designed for code summarization tasks specifically, thus provides better performance as our re-sults show in section 5 foot_0 . furthermore, our proposed approach can be combined with any pretrained models, other than bert, potentially improving upon their original performances. ## proposed approach as shown in figure 2 , our proposed model has a common encoder-decoder architecture, consisting of training two different but similar tasks: 1) code to summary generation, and 2) api sequence to summary generation. although we trained two different tasks, our main task is code to summary generation. we first describe the encoder-decoder architecture, followed by the multi-task learning framework of the two tasks. ## experimental setup this section describes the datasets used in our experiments (section 4.1), the different metrics used in the automatic evaluation (section 4.2), and the qualitative evaluation (section 4.3). the different baselines are discussed in section 4.4 and the hyperparameters to our models are listed in section 4.5. ## results we provide the automatic evaluation of the baseline models and our proposed model in section 5.1 followed by the human evaluation under section 5.2. in addition to the human evaluation, we also provide several examples for additional qualitative analysis. ## conclusions in this work, we proposed a novel and effective multi-task approach for generating summaries from code. two different but similar tasks, 1) generating summaries from code, and 2) generating summaries from api sequence, are trained simultaneously. our proposed model also considers modeling every line of code (loc) whereas existing work treats code as a single contiguous sequence. to the best of our knowledge, this is the first work that utilizes a natural language-based pre-trained language model for a code summarization task. our experimental results on two popular datasets, java and python, show that our proposed model performs better than all baselines, achieving the new state-ofthe-art performances. additionally, both the multitask component and loc modeling component of our proposed model are demonstrated to be effective. furthermore, our proposed approach can be combined with any pre-trained models, other than bert, potentially improving upon their original performances.