Boosting Code Summarization by Embedding Code Structures (2022)

Abstract: Recent research on code summarization relies on the structural information from the abstract syntax tree (AST) of source codes. It is, however, questionable whether it is the most effective to use AST for expressing the structural information. We find that a program dependency graph (PDG) can represent the structure of a code more effectively. We propose PDG Boosting Module (PBM) that encodes PDG into graph embedding and the framework to implement the proposed PBM with the existing models. PBM achieves improvements of 6.67% (BLEU) and 7.47% (ROUGE) on average. We then analyze the experimental results, and examine how PBM helps the training of baseline models and its performance robustness. For the validation of robustness, we measure the performance of an out-of-domain benchmark dataset, and confirm its robustness. In addition, we apply a new evaluation measure, SBERT score, to evaluate the semantic performance. The models implemented with PBM improve the performance of SBERT score. This implies that they generate summaries that are semantically more similar to the reference summary.

## introduction in the early stage of generating code summaries, researchers adopted information retrieval techniques @xcite @xcite to capture source code semantics. however, code summaries produced by such techniques are often inaccurate to use in practice @xcite . with the help of deep learning, researchers proposed the neural machine translation (nmt) frameworks that automatically produce summaries from source code @xcite @xcite . rather than using code sequence as a sole input, several models built upon the transformer @xcite used additional data structures and information @xcite @xcite to learn obscure features that otherwise would be discarded. recent studies use pretrained models and implementations of graph structures to improve the performance of code summarization. pretrained models are built upon training a huge quantity of benchmark datasets within a long period. code-bert @xcite and codet5 @xcite are popular pretrained models for various code-related tasks. the graph embedding is regarded as effective on providing code semantics; especially abstract syntax tree (ast) is the most popular supplement types for reflecting the hierarchical structure of codes. several researchers used asts and improved the performance of source code summarization @xcite @xcite . furthermore, combining with the transformer, there are a few improved models including mast+gcn @xcite , basts @xcite , cast @xcite and sit @xcite . while asts are widely used to capture code structure information, they cannot capture global information between tokens well due to the deep depths @xcite @xcite . thus, recent researches consider graphs other than asts to capture structural information. @xcite pretrain the model after dividing the code based on control flow graph (cfg), and @xcite proposed a method of capturing the global structure by learning the data flow relationship between variables. @xcite utilize a new type of graph called code property graph @xcite , which combines cfg and ast, and propose a hybrid gnn using cpg. when models utilize ast as auxiliary information, structural information is treated and delivered by token level representation. many researchers are performed through extracting structural information from the token representations. however, such token representations of asts fail to provide semantics of statements and predicates @xcite . we perform code summarization by capturing the structural information through a program dependency graph (pdg) to solve such problem. we implement the encoder module that takes pdgs of source codes as inputs to several baseline models, and evaluate the improvement from our graph module for the summary generation performance of each model. baseline models we perform experiments are sit @xcite that applies ast on a transformer and codebert @xcite , including the transformer @xcite . the experimental results of our implementation show the performance improvements of average corpus-bleu 6.67% and rouge-l 7.47%. however, these improvements are not sufficient to demonstrate that our implementation accurately captures structural information. thus, we further ask the following questions based on the initial experimental results. rq1: is the structural code information such as graph embedding indeed helpful for generating a code summary? rq2: what difference does the proposed graph structure have compared to the popularly used ast and what is better between two graph structures? rq3: does the proposed model show the robust performance for out-of-domain data? our code is available at https://github. com/sjk0825/coling2022 . ## methodology we propose pbm (pdg boosting module) which improves the capability of capturing the structure information by embedding pdg to the encoder. in this section, we explain what pdg is and then, show how our pbm embeds pdg. finally, we demonstrate the implementation of our pbm with the baseline models of the code summarization task. we illustrate the overview of the framework implementation of pbm in figure 2 . our module applies the pdg to improve the baseline models but for better analysis, we also develop a module for asts that act the same as our pbm and show the experimental results in section 4. ## experiments and analysis our experiment uses two benchmark datasets for the code summarization task. the first is ccsd @xcite which is the dataset of c programs and the next is tl-codesum @xcite the evaluation metrics are corpus-bleu @xcite and rouge-l @xcite score that are widely used for the verification of code summarization performance. we denote them as bleu and rouge. additionally, we use sbert @xcite score to address the semantic performance that cannot be measured by the prior two metrics. as we apply pbm to baseline models, one experiment is to compare the performance improvements by adding pbm to baseline models. another experiment is to check which graph type is more suitable for our task. for the second question, we consider two graph types, ast and pdg, each of which is constructed from data via joern foot_0 and srcml foot_1 . ## conclusions recently, there have been several researches for improving the code summarization @xcite @xcite . one approach is to use the structural information of source code by extracting its ast (abstract syntax tree) @xcite @xcite . while this approach works better than the one without asts, it turns out that a model with asts cannot capture the global structure owing to deep depth structure @xcite @xcite . we have studied the limit of this approach and suggested a new module pbm that utilize pdgs containing the information of control and data dependencies. we have observed that pbm improves the performance of baseline models; pbm captures the structural information of source codes in a statement level. since bleu and rouge are computed by matching tokens of the reference summary, we have considered another metric, sbert score, for measuring the semantic difference to fully analyze the effectiveness of pbm; sbert score evaluates the semantic similarity between sentences. the experimental results have showed that pbm achieves the performance increase in capturing the semantics of source code in the sbert score as well. we noticed that the benchmark dataset of source codes and summaries can be vulnerable in generalization of a model. thus, we have ran additional experiments using an out-domain test dataset of ccsd, and confirmed that our pbm is effective for the ood case as well. for future directions, we aim to evaluate the generalizability of the code summarization models more precisely. in addition to the out-domain dataset of ccsd, we plan to evaluate pbm with other benchmark datasets for the ood measurements. furthermore, we plan to design a model that has a consistently good performance on generating code summaries for different programming languages.