Modeling Hierarchical Syntax Structure with Triplet Position for Source Code Summarization (2022)

Abstract: Automatic code summarization, which aims to describe the source code in natural language, has become an essential task in software maintenance. Our fellow researchers have attempted to achieve such a purpose through various machine learning-based approaches. One key challenge keeping these approaches from being practical lies in the lacking of retaining the semantic structure of source code, which has unfortunately been overlooked by the state-of-the-art. Existing approaches resort to representing the syntax structure of code by modeling the Abstract Syntax Trees (ASTs). However, the hierarchical structures of ASTs have not been well explored. In this paper, we propose CODESCRIBE to model the hierarchical syntax structure of code by introducing a novel triplet position for code summarization. Specifically, CODESCRIBE leverages the graph neural network and Transformer to preserve the structural and sequential information of code, respectively. In addition, we propose a pointer-generator network that pays attention to both the structure and sequential tokens of code for a better summary generation. Experiments on two real-world datasets in Java and Python demonstrate the effectiveness of our proposed approach when compared with several state-of-the-art baselines.

## introduction code documentation in the form of code comments has been an integral component of software development, benefiting software maintenance @xcite , code categorization @xcite and retrieval @xcite . however, few real-world software projects are welldocumented with high-quality comments. many projects are either inadequately documented due to missing important code comments or inconsistently documented due to different naming conven-tions by developers, e.g., when programming in legacy code bases, resulting in high maintenance costs @xcite . therefore, automatic code summarization, which aims to generate natural language texts (i.e., a short paragraph) to describe a code fragment by extracting its semantics, becomes critically important for program understanding and software maintenance. recently, various works have been proposed for code summarization based on the encoder-decoder paradigm, which first encodes the code into a distributed vector, and then decodes it into naturallanguage summary. similarly, several works @xcite proposed to tokenize the source code into sequential tokens, and design rnn and cnn to represent them. one limitation of these approaches is that they only consider the sequential lexical information of code. to represent the syntax of code, several structural neural networks are designed to represent the abstract syntax trees (ast) of code, e.g., @xcite , tbcnn @xcite , and graph neural networks (gnns) @xcite . to further improve the efficiency on ast representation, various works @xcite proposed to linearize the asts into a sequence of nodes or paths. despite much progress on code summarization, there are still some limitations in code comprehension for generating high-quality comments. particularly, when linearizing the asts of code into sequential nodes or paths, the relationships between connected nodes are generally discarded. although the gnn-based approaches can well preserve the syntax structure of code, they are insensitive to the order of nodes in ast. for example, given the expressions @xmath0, current approaches cannot capture the orders of variables b and c. however, these orders are critical to accurately preserve the semantics of code. to address the aforementioned limitation, this @xcite a', @xmath1, @xmath2, @xmath3, @xmath4, @xmath5, @xmath6, inspired by the positional encoding used in sequence modeling @xcite , and incorporates the triplet position into current gnns for better code summarization. the triplet position records the depth, width position of its parent, and width position among its siblings for each node. to utilize the triplet position in ast, this paper proposes codescribe, an encoder-decoderbased neural network for source code summarization. specially, we initialize the embedding of each ast node by incorporating the triplet positional embeddings, and then feed them into an improved gnn, i.e., graphsage @xcite to represent the syntax of code. in addition, we also account for the sequential information of code by using a transformer encoder @xcite . in such a case, the decoding process is performed over the learned structural features of ast and sequential features of code tokens with two multi-head attention modules. to generate summaries with higher quality, we further design a pointer-generator network based on multi-head attention @xcite , which allows the summary tokens to be generated from the vocabulary or copied from the input source code tokens and asts. to validate the effectiveness of our proposed codescribe, we conduct experiments on two real-world datasets in java and python. overall, the primary contributions of this paper are as follows. ## hierarchical syntax in triplet position recent studies have showed promising results by using ast context for tasks based on code representation learning @xcite @xcite . therefore, our work also relies on ast information besides source code tokens. as a type of intermediate representation, ast represents the hierarchical syntactic structure for source code, which is an ordered tree with labeled nodes (cf. figure 1 ). in this work, we divide the nodes into two categories: (1) function node that controls the structure of ast and function realization, e.g., module and assign in figure 1 , and (2) attribute node that provides the value or name of its parent function node, which is always visualized as leaf node, such as 'a' and 'b' in dotted boxes of figure 1 . due to the strict construction rules of ast, positions are crucial for ast nodes. for example in figure 1 , the node binop has two children with the same label name. if the positions of the two siblings are swapped, the source code will become @xmath7, which is totally different from the intent of the code @xmath8. however, gnns are insensitive to the positions of neighbouring nodes when encoding such tree structures. based on this obser- probabilities of next summary tokens 6× multi-head att with res. & norm. feed forward with res. & norm. source code & ast embedding 2× source code masked multi-head att multi-head att. with res. & norm. sum. embedding pointer-generator 6× summary tokens multi-head att. with res. & norm. gnn (graphsage) ast feed forward with res. & norm. multi-head att multi-head att. relu res. & norm. encoders decoder multi-source pointer-generator network positional embeddings + + triplet positional embeddings positional embeddings + figure 2: the architecture of codescribe model. att., res., and norm. denote attention, residual connection, and layer normalization, respectively. taking a python code snippet @xmath9, figure 1 illustrates its ast structure with triplet positions of nodes. specifically, by traversing the tree, we can represent the function node (name,{2,0,0}) as the first child node of node (assign,{1,0,0}): the depth position 2 means the third level (counting from the top to bottom starting with 0); the second width position 0 means that the parent node assign is the first function node at this level (counting from the left to right); and the third position 0 indicates that the node is the first (counting from left to right) among its siblings (i.e., all children nodes of node assign). another example is the node ('a',{3,0,-1}). the difference lies in the third position that represents it is an attribute node and it is the first among the siblings. in particular, we set the position of root node module to {0,0,0} as it has no parent node. this triplet positioning is very precise and unique, allowing to track and discriminate among the name nodes which also include (name,{3,1,0}) and (name,{3,1,2}). ## experiments we conduct experiments to answer the following research questions: (1) how effective is code-scribe compared with the state-of-the-art baselines? (2) how effective is the structure design of codescribe? (3) what is the impact of model size on the performance of codescribe? we also perform a qualitative analysis of two detailed examples. ## related work with the development of deep learning, most works have considered code summarization as a sequence generation task. in many of the recent approaches, source code snippets are modeled as plain texts based on rnns @xcite @xcite . for example, @xcite proposed an rnn-based model that learns api knowledge from a different but related task and incorporates the knowledge into code summarization. wei et al. (2019) presented a dual learning framework based on lstms to train code generation and code summarization and improve the performances of both tasks. ye et al. (2020) combined code summarization and code generation to train the code retrieval task via multi-task learning, which achieved competitive performance for the code summarization task. most recently, ahmad et al. (2020) applied transformer to encoding the source code sequence to improve the summarization performance. since considering source code as plain text ignores the structural information in code, recent works have explored the ast of code and modeled the tree-based structure for code summarization. typically, hu et al. (2018a) proposed a structurebased traversal (sbt) method to traverse asts into node sequences and used a sequence-to-sequence model based on lstms to generate code comments. alon et al. (2019) represented a code snippet as a set of compositional paths in its ast and used lstms to encode these paths. shido et al. (2019) extended tree-lstm (tai et al., 2015) to multi-way tree-lstm to learn the representation of ast for code summary generation. liu et al. (2020) built code property graph (cpg) @xcite based on ast and combined retrieval method and gnns for describing c programming language. the latest work @xcite performed graph convolutional networks (gcns) @xcite before transformer framework to learn ast representation for summary generation. to represent the code comprehensively, more and more works have paid attention to both the source code and the ast for code summarization. for example, @xcite integrated both ast node sequence and source code into a hybrid learning framework based on grus. wei et al. (2020) and zhang et al. (2020) both utilized the information retrieval techniques to improve the quality of code summaries that are generated from the code snippets and asts. wan et al. (2018) incorporated ast as well as sequential content of code snippet into a deep reinforcement learning framework based on lstm and ast-based lstm. leclair et al. (2020) proposed a graph-based neural architecture for code summarization, which uses grus and gcn to encode ast and grus to learn source code sequence. wang et al. (2022) presented the first hierarchical-attention based learning approach for code summarization by integrating source code, type-augmented ast, and control-flow graphs. recently, several pre-trained models, e.g., code-bert @xcite , codet5 @xcite , plbart @xcite and co-text @xcite , have been proposed to better represent the source code, and verified on code summarization. for example, codebert @xcite ) is a pre-trained model based on elec-tra @xcite , which has achieved promising performance on downstream tasks including code summarization. codet5 @xcite considers the token type information in code and builds on the t5 architecture @xcite that utilizes denoising sequence-to-sequence pre-training. plbart @xcite is another start-of-the-art pre-trained model on an extensive collection of python and java functions, as well as their natural language summaries via denoising auto-encoding. note that, our work is aim to introduce an encoder network with a novel triplet position to better represent the hierarchical structure of programs, rather than pre-training a language model for source code. we think that our introduced encoder can be easily incorporated into the pre-training models through masking and predicting code tokens or code graphs. we leave the comparison between our model and those mentioned pre-trained code models to future work. ## conclusion this paper has presented codescribe, an encoder-decoder-based neural network for source code summarization. codescribe designs a triplet position to model the hierarchical syntax structure of code, which is then incorporated into the transformer and gnn based framework for better representation of lexical and syntax information of code, respectively. the performance of codescribe is further enhanced by the introduced multi-source pointer generator in decoding. experiments on two benchmarks reveal that the summaries generated by codescribe are of higher quality when compared with several recent state-of-the-art works. wei ye, rui xie, jinglei zhang, tianxiang hu, xiaoyin wang, and shikun zhang. 2020. leveraging code generation to improve code retrieval and summarization via dual learning. in proceedings of the web conference 2020, pages 2309-2319. jian zhang, xu wang, hongyu zhang, hailong sun, and xudong liu. 2020. retrieval-based neural source code summarization. in icse '20: 42nd international conference on software engineering, pages 1385-1397. jian zhang, xu wang, hongyu zhang, hailong sun, kaixuan wang, and xudong liu. 2019. a novel neural source code representation based on abstract syntax tree. in proceedings of the ieee/acm 41st international conference on software engineering, pages 783-794.