R ETRIEVAL -A UGMENTED
G ENERATION
FOR
C ODE
S UMMARIZATION VIA H YBRID GNN
Anonymous authors
Paper under double-blind review
A BSTRACT
Source code summarization aims to generate natural language summaries from structured code snippets for better understanding code functionalities.
However, automatic code summarization is challenging due to the complexity of the source code and the language gap between the source code and natural language summaries.
Most previous approaches either rely on retrieval-based (which can take advantage of similar examples seen from the retrieval database, but have low generalization performance) or generation-based methods (which have better generalization performance, but cannot take advantage of similar examples).
This paper proposes a novel retrieval-augmented mechanism to combine the beneﬁts of the both worlds.
Furthermore, to mitigate the limitation of Graph Neural Networks (GNNs) on capturing global graph structure information of source code, we propose a novel attention-based dynamic graph to complement the static graph representation of the source code, and design a hybrid message passing GNN for capturing both the local and global structural information.
To evaluate the proposed approach, we release a new challenging benchmark, crawled from diversiﬁed large-scale open-source
C projects (total 95k+ unique functions in the dataset).
Our method achieves the state-of-the-art performance, improving existing methods by 1.65 , 1.76 and 1.
in terms of BLEU-4, ROUGE-L and METEOR.
I NTRODUCTION
With software growing in size and complexity, developers tend to spend nearly 90% (Wan et al., 2018) effort on software maintenance ( e.g ., version iteration and bug ﬁx) in the completed life cycle of software development.
Source code summary, in the form of natural language, plays a critical role in comprehension and maintenance process and greatly reduces the effort of reading and comprehending programs.
However, manually writing code summaries is tedious and time-consuming, and with the acceleration of software iteration, it has become a heavy burden for software developers.
Hence, source code summarization which automates concise descriptions of programs is meaningful.
Automatic source code summarization is a crucial yet far from settled problem.
The key challenges include: 1) the source code and the natural language summary are heterogeneous, which means they may not share common lexical tokens, synonyms, or language structures and 2) the source code is complex with complicated logic and variable grammatical structure, making it hard to learn the semantics.
Conventionally, information retrieval (IR) techniques have been widely used in code summarization (Eddy et al., 2013;
Haiduc et al., 2010; Wong et al., 2015; 2013).
Since code duplication (Kamiya et al., 2002; Li et al., 2006) is common in “big code” (Allamanis et al., 2018), early works summarize the new programs by retrieving the similar code snippet in the existing code database and use its summary directly.
Essentially, the retrieval-based approaches transform the code summarization to the code similarity calculation task, which may achieve promising performance on similar programs, but are limited in generalization, i.e .
they have poorer performance on programs that are very different from the code database.
To improve the generalization performance, recent works focus on generation-based approaches.
Some works explore Seq2Seq architectures (Bahdanau et al., 2014;
Luong et al., 2015) for generate summaries from the given source code.
The Seq2Seq-based approaches (Iyer et al., 2016; Hu et al., 2018a; Alon et al., 2018) usually treat the source code or abstract syntax tree parsed from the source code as a sequence and follow a paradigm of encoder-decoder with attention mechanism for generating a summary.
However, these works only rely on sequential models, which are struggling to capture the rich semantics of source code e.g ., control dependencies and data dependencies.
In addition, generation-based approaches typically cannot take advantage of similar examples from the retrieval database, as retrieval-based approaches do.
To better learn the semantics of the source code, Allamanis et al.
(Allamanis et al., 2017) lighted up this ﬁeld by representing programs as graphs.
Some follow-up works (Fernandes et al., 2018) attempted to encode more code structures ( e.g ., control ﬂow, program dependencies) into code graphs with graph neural networks (GNNs), and achieved the promising performance than the sequencebased approaches.
Existing works (Allamanis et al., 2017; Fernandes et al., 2018) usually convert code into graph-structured input during preprocessing, and directly consume it via modern neural networks ( e.g ., GNNs) for computing node and graph embeddings.
However, most GNN-based encoders only allow message passing among nodes within a k -hop neighborhood (where k is usually a small number such as 4) to avoid over-smoothing (Zhao & Akoglu, 2019; Chen et al., 2020), thus capture only local neighborhood information and ignore global interactions among nodes.
Even there are some works (Li et al., 2019) that try to address this challenging with deep GCNs (i.e.,layers) (Kipf & Welling, 2016) by the residual connection (He et al., 2016), however, the computation cost cannot endure in the program especially for a large and complex program.
For example, on our benchmark, the average/max node size of functions are 70/200 and the average node degree is 1.77.
To address these challenges, we propose a framework for automatic code summarization, namely
Hybrid-GNN (HGNN) .
Speciﬁcally, from the source code, we ﬁrst construct a code property graph (CPG) based on abstract syntax tree (AST) with different types of edges ( i.e ., Flow To, Reach).
In order to combine the beneﬁts of both retrieval-based and generation-based methods, we propose a novel retrieval-based augmentation mechanism to retrieve the source code that is most similar to the current program from the retrieval database (excluding the current program itself), and add the retrieved code as well as the corresponding summary as auxiliary information for training the model.
In order to go beyond local graph neighborhood information, and capture global interactions in the program, we further propose an attention-based dynamic graph by learning global attention scores ( i.e ., edge weights) in the augmented static CPG.
Then, a hybrid message passing (HMP) is performed on both static and dynamic graphs.
We also release a new code summarization benchmark by crawling data from popular and diversiﬁed projects containing 95k+ functions in C programming language and make it public 1 .
We highlight our main contributions as follows:
• We propose a general-purpose framework for automatic code summarization, which combines the beneﬁts of both retrieval-based and generation-based methods via a novel retrieval-based augmentation mechanism.
• We innovate a Hybrid-GNN by fusing the static graph (based on code property graph) and dynamic graph (via structure-aware global attention mechanism) to mitigate the limitation of the GNN on capturing global graph information.
• We release a new challenging C benchmark for the task of source code summarization.
• We conduct an extensive experiment to evaluate our framework.
The proposed approach achieves the state-of-the-art performance and improves existing approaches by 1.65 , 1.76 and 1.81 in terms of BLEU-4, ROUGE-L and METEOR metrics.
H YBRID -GNN F RAMEWORK
In this section, we introduce the proposed framework Hybrid-GNN (HGNN) , as shown in Figure 1, which mainly includes four components: 1) Retrieval-augmented Static Graph Construction (c.f.,
Section 2.2), which incorporates retrieved code-summary pairs to augment original code for learning.
2) Attention-based Dynamic Graph Construction ( c.f., Section 2.3), which allows message passing among any pair of nodes via a global attention mechanism.
3) HGNN , ( c.f., Section 2.4), which incorporates information from both static graphs and dynamic graphs with Hybrid Message Passing.
4) Decoder ( c.f., Section 2.5), which utilizes an attention-based LSTM (Hochreiter & Schmidhuber, 1997) model to generate a summary.
Retrieved Summary
SourceCode
RetrievalCPG
Aug
Static Message Passing
A X( + … + )
Dynamic Message Passing
Hybrid
Decoder
GeneratedSummary
Hybrid GNN Retrieval-augmented Graph
SummaryEncoding
Graph Embedding
GRU Updates
Node Embeddings
Attention-based Graph
RetrievalDatabase
SourceCPG
Aggregation
2. P ROBLEM F ORMULATION
In this work, we focus on generating summaries for the given functions (Wan et al., 2018; Zhang et al., 2020).
We deﬁne a dataset as D = { ( c, s ) | c ∈ C, s ∈ S } , where c is the source code of a function in the function set C and s represents its targeted summary in S .
The task of code summarization is, given a source code c , to generate the best summary consisting of a sequence of tokens ˆ s = ( t 1 , t 2 , ..., t T ) which maximizes the conditional likelihood ˆ s = argmax s P ( s | c ) .
In this paper, we follow the problem setting and propose the technique to learn the mapping from the source code to the natural language summary.
2. R ETRIEVAL - AUGMENTED S TATIC G RAPH 2.2. G RAPH I NITIALIZATION
The source code of a function can be represented as Code Property Graph (CPG) (Yamaguchi et al., 2014), which is built on the abstract syntax tree (AST) with different type of edges ( i.e ., Flow To,
Control, Deﬁne/Use, Reach).
Formally, one raw function c could be represented by a multi-edged graph g ( V , E ) , where V is the set of AST nodes, ( v
, u ) ∈
E denotes the edge between the node v and the node u .
A node v consists of two parts: the node sequence and the node type .
An illustrative example is shown in Figure 2.
For example, in the red node, a %2 =
=
is the node sequence and
Condition is the node type.
An edge ( v, u ) has a type, named edge type , e.g ., AST type and Flow
To type.
For more details about the CPG, please refer to Appendix A.
Initialization Representation.
Given a CPG, we utilize a BiLSTM to encode its nodes.
We represent each token of the node sequence, each node type and each edge type using the learned embedding matrix E seqtoken , E nodetype and E edgetype , respectively.
Then nodes and edges of the CPG can be encoded as:
[Equation]
where l is the number of tokens in the node sequence of v .
For the sake of simplicity, in the following section, we use h v and e v,u to represent the embedding of the node v and the edge ( v, u ) , respectively, i.e ., encode _ node ( v ) and encode _ edge ( v, u ) .
Given the source code c of a function as well as the
CPG g ( V , E ) , H c ∈ R m ×
d denotes the initial node matrix of the CPG, where m is the total number of nodes in the CPG and d is the dimension of the node embedding.
2.2. R ETRIEVAL - BASED A UGMENTATION
While retrieval-based methods can perform reasonably well on examples that are similar to those examples from a retrieval database, they typically have low generalization performance and might perform poorly on dissimilar examples.
On the contrary, generation-based methods usually have better generalization performance, but cannot take advantage of similar examples from the retrieval database.
[Figure]
CFGEntryNode example ()
FunctionDef int a = rand( ) IdentiﬁerDeclStatement a % 2 ==
Condition int b = a ++ IdentiﬁerDeclStatement
EXIT CFGExitNode call(b) ExpressionStatement
ParameterList void
[Figure]
int IdentiﬁerDeclType a Identiﬁer a = rand ( ) AssignmentExpr rand ( ) CallExpression a % 2 ==
EqualityExpression a % 2 MultiplicativeExpression a Identiﬁer 2 PrimaryExpression
PrimaryExpression int IdentiﬁerDeclType b Identiﬁer b = a ++ AssignmentExpr a ++ IncDecOp ++ IncDec a Identiﬁer
[Figure]
call Callee b Argument b Identiﬁer b Symbol
[Figure]
Identiﬁer
[Figure]
In this work, we propose to combine the beneﬁts of the two worlds, and design a retrieval-augmented generation framework for the task of code summarization.
In principle, the goal of code summarization is to learn a mapping from source code c to the natural language summary s = f ( c ) .
In other words, for any source code c (cid:48) , a code summarization system can produce its summary s (cid:48)
=
f ( c (cid:48) ) .
Inspired by this observation, conceptually, we can derive the following formulation s = f ( c ) − f
( c (cid:48) )
+ s (cid:48) .
This tells us that we can actually compute the semantic difference between c and c (cid:48) , and further obtain the desired summary s for c by considering both the above semantic difference and s (cid:48) which is the summary for c (cid:48) .
Mathmatically, our goal becomes to learn a function which takes as input c , c (cid:48) and s (cid:48) and outputs the summary s for c , that is, s = g ( c, c (cid:48) , s (cid:48) ) .
This motivates us to design our Retrieval-based Augmentation mechanism, as detailed below.
Step 1: Retrieving.
For each sample ( c, s ) ∈ D , we retrieve the most similar sample: ( c (cid:48) , s (cid:48) )
=
argmax ( c (cid:48) ,s (cid:48) )
∈ D (cid:48) sim ( c, c (cid:48) ) , where c (cid:54) =
c (cid:48) , D (cid:48) is a given retrieval database and sim ( c, c (cid:48) ) is the text similarity.
Following Zhang et al.
(2020), we utilize Lucene for retrieval and calculate the similarity score z between the source code c and the retrieved code c (cid:48) by dynamic programming (Bellman, 1966)
z = 1 − dis ( c,c (cid:48) ) max ( | c | , | c (cid:48) | ) , where dis ( c, c (cid:48) ) is the text edit distance
Step 2: Retrieved Code-based Augmentation.
Given the retrieved source code c (cid:48) for the current sample c , we adopt a fusion strategy to inject retrieved semantics into the current sample.
The fusion strategy is based on their initial graph representations ( H c and H c (cid:48) ) with an attention mechanism:
• To capture the relevance between c and c (cid:48) we design an attention function, which computes the attention score matrix A aug based on the embedding of each pair of nodes in CPGs of c and c (cid:48) :
[Equation]
where W ∈ R d × d is the matrix with d -dim embedding size and ReLU is the rectiﬁed linear unit.
• Multiply
the attention matrix A aug with the retrieved representation H c (cid:48) to inject retrieved features into H c :
[Equation]
where z is the similarity score, which is introduced to weaken the negative impact of c (cid:48) on the original training data c , i.e ., when the similarity of c and c (cid:48) is low.
• Finally, we merge H (cid:48) c and the original H c to get the ﬁnal representation of c .
[Equation]
where W c , W (cid:48)
c
∈ R
d × d are weighted matrices and comp is the retrieval augmented node representation.
Step 3:
Retrieved Summary-based Augmentation.
We further encode the retrieved summary s (cid:48) with another BiLSTM model. We represent each token t (cid:48) i of s (cid:48) using the learned embedding matrix
E seqtoken .
Then s (cid:48) can be encoded as: [Equation] where h t (cid:48) i is the state of the BiLSTM model for the token t (cid:48)
i in s (cid:48) and T is the length of s (cid:48) .
We also multiply the similarity score z to [ h t (cid:48) 1 , ..., h t (cid:48) T ] and concatenate with the graph encoding results (i.e., the outputs of the GNN encoder) as the input [GNN output , z h t (cid:48) 1 , ..., z h t (cid:48) T ] to the decoder.
2. A TTENTION - BASED D YNAMIC G RAPH
Due to that GNN-based encoders usually consider the k -hop neighborhood, the global relation among nodes in the static graph (see Section 2.2.1) may be ignored.
In order to better capture the global semantics of source code, based on the static graph, we propose to dynamically construct a graph via structure-aware global attention mechanism, which allows message passing among any pair of nodes.
The attention-based dynamic graph can better capture the global dependency among nodes, and thus supplement the static graph.
Structure-aware Global Attention.
The construction of the dynamic graph is motivated by the structure-aware self-attention mechanism proposed in Zhu et al.
Given the static graph, we compute a corresponding dense adjacency matrix A dyn based on a structure-aware global attention mechanism and the constructed graph namely attention-based dynamic graph .
Unlike the selfattention mechanisms in Zhu et al.
(2019), we consider not only the node semantics but also the edges in the static graph ( i.e ., the CPG of the training data) when computing attention scores between any pair of nodes.
[Equation]
where h v , h u ∈ comp are the augmented node embedding for any node pair ( v, u ) in the CPG.
Note that the global attention considers each pair of nodes of the CPG, regardless of whether there is an edge between them.
e v,u ∈ R d e is the edge embedding and W Q , W K ∈ R d × d ,
W R ∈ R
d e × d are parameter matrices, d e and d are the dimensions of edge embedding and node embedding, respectively.
The adjacency matrix A dyn normalizes with softmax function, which will be used to compute dynamic message passing (see Section 2.4).
[Equation]
2. H YBRID GNN
To better incorporate the information of the static graph and the dynamic graph, we propose the
Hybrid Message Passing (HMP), which are performed on both retrieval-augmented static graph and attention-based dynamic graph.
Static Message Passing.
We incorporate the edge type embedding to encode the static graph.
For every node v at each computation hop k , we apply an aggregation function to calculate the aggregated vector h kv by consdering a set of neighboring node embeddings computed from the previous hop.
[Equation]
where W P ∈ R d e × d is the weighted matrix and N ( v ) is a set of the neighboring nodes which are directly connected with v .
For each node v , h 0 v is the initial augmented node embedding of v , i.e ., h v ∈ comp .
Dynamic Message Passing.
The node information and edge information are propagated on the attention-based dynamic graph with the adjacency matrices A dyn , deﬁned as
[Equation]
where m is the total number of nodes, v j is the j th node,
W V ∈ R d × d , W F ∈ R d
e × d are learned matrices, and e v,v j is the edge embedding between the edge v, v j .
Similarly, h (cid:48)
v j is the initial augmented node embedding of v in comp .
Hybrid Message Passing.
Given the static/dynamic aggregated vectors h kv / h (cid:48)
kv for static and dynamic graphs, we fuse both vectors and feed the resulting vector to a Gated Recurrent Unit (GRU) to update node representations.
[Equation]
where f 0 v is the augmented node initialization in comp .
The fusion function Fuse is designed as a gated sum of two inputs.
[Equation]
[Equation] where (cid:12) is the component-wise multiplication, σ is a sigmoid function and z is a gating vector.
After n hops of GNN computation, we obtain the ﬁnal node representation f nv and then apply max-pooling over all nodes { f nv |∀ v ∈ V} to get the graph representation.
2. D ECODER
The decoder is similar with other state-of-the-art Seq2seq models (Bahdanau et al., 2014; Luong et al., 2015) where an attention-based LSTM decoder is used.
The decoder takes the input of the concatenation of the node representation and the representation of the retrieved summary ( i.e ., s (cid:48)
=
( t (cid:48) 1 , ..., t (cid:48) T ) ):
[ f nv 1 ; ... ; f nv m ; z h t (cid:48) 1 ; ... ; z h t (cid:48) T ] , where n is the number of hops and m is the number of nodes in the CPG.
The initial hidden state of decoder is the fusion of graph representation and the weighted (i.e., multiply similarity score z ) ﬁnal state of retrieved summary.
We train the model with regular cross-entropy loss, deﬁned as L = (cid:80) t 1 − log P ( s ∗ t | c, s ∗ <t ) , where s ∗ t is the word at the t -th position of the ground-truth output and c is the source code of the function.
To alleviate the exposure bias, we utilize schedule teacher forcing (Bengio et al., 2015).
During the inference, we use beam search to generate ﬁnal results.
E XPERIMENTS 3. S ETUP
We evaluate our proposed framework against a number of state-of-the-art methods.
Speciﬁcally, we classify the selected baseline methods into three groups: 1) Retrieval-based approaches: TFIDF (Haiduc et al., 2010) and NNGen (Liu et al., 2018), 2) Sequence-based approaches: CODENN (Iyer et al., 2016; Barone & Sennrich, 2017), Transformer (Ahmad et al., 2020), HybridDRL (Wan et al., 2018), Rencos (Zhang et al., 2020) and Dual model (Wei et al., 2019), 3)
Graphbased approaches: SeqGNN (Fernandes et al., 2018).
In addition, we implemented two another graph-based baselines:
GCN2Seq and GAT2Seq, which respectively adopt the Graph Convolution (Kipf & Welling, 2016) and Graph Attention (Velickovic et al., 2018) as the encoder and a LSTM as the decoder for generating summaries.
Note that Rencos (Zhang et al., 2020) combines the retrieval information into Seq2Seq model, we classify it into Sequence-based approaches.
More detailed description about baselines and the conﬁguration of HGNN can be found in the Appendix B and C.
Existing benchmarks (Barone & Sennrich, 2017; Hu et al., 2018b) are all based on high-level programming language i.e ., Java, Python.
Furthermore, they have been conﬁrmed to have extensive duplication, making model overﬁt to the training data that overlapped with the testset Fernandes et al.
; Allamanis (2019).
We are the ﬁrst to explore neural summarization on C programming language and make our benchmark public to beneﬁt the academia and industry.
We crawled from popular C repositories on GitHub and extract function-summary pairs based on the documents of functions.
After a strict deduplication process, we kept 95k+ unique function-summary pairs and name it C Code Summarization Dataset (CCSD).
To further test the model generalization ability, we construct in-domain functions and out-of-domain functions by dividing the projects into two sets, denoted as a and b .
For each project in a , we randomly select some of functions in this project as the training data and the unselected functions are the in-domain validation/test data.
All functions in projects b are regarded as out-of-domain test data.
Finally, we obtain 82,656 training functions, 4,340 in-domain validation functions, 4,124 in-domain test functions and 2,264 out-of-domain test functions.
For the retrieval augmentation, we also use the training set as the retrieval database, i.e .,
D (cid:48)
=
D (see Step 1 in Section 2.2).
More details about data processing, please refer to Appendix D.
Similar to previous works (Zhang et al., 2020; Wan et al., 2018; Fernandes et al., 2018; Iyer et al., 2016), BLEU (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005) and ROUGE-L (Lin, 2004) are used as our automatic evaluation metrics.
These metrics are popular in machine translation, text summarization.
Except for these similarity-based metrics, we also conduct a human evaluation study to evaluate semantic similarity.
We invite 5 Ph.D students and 10 master students as the volunteers, who have rich C programming experiences.
The volunteers are asked to rank summaries generated from the anonymized approaches from 1 to 5 ( i.e ., 1: Poor, 2: Marginal, 3: Acceptable, 4: Good, 5: Excellent) based on the relevance of the generated summary to the source code and the degree of similarity between the generated summary and the actual summary.
Speciﬁcally, we randomly [Table] choose 50 functions for per approach with the corresponding generated summaries and ground-truths.
After the summarizes are ranked [Table] we calculate the average score for each function.
Higher scores mean better quality
[Table]
3. C OMPARISON WITH THE BASELINES
Table 1 shows the evaluation results including two parts: the comparision with baselines and the ablation study.
Consider the comparison with state-of-the-art baselines, in general, we ﬁnd that our proposed model outperforms existing methods by a signiﬁcant margin on both in-domain and out-of-domain datasets [Table] and shows good generalization performance.
Compared with others, on in-domain dataset, the simple retrieval-based approaches could achieve competitive performance on BLEU-4, however ROUGE-L and METEOR are fare less than ours.
They also do not perform well on out-of-domain dataset.
Even without augmentation ( HGNN w/o augment ), our approach still outperforms the graph-based approaches ( i.e ., GCN2Seq, GAT2Seq and SeqGNN), which further demonstrates the effectiveness of Hybrid-GNN for additionally capturing global graph information.
Compared with Sequence-based approaches, HGNN w/o augment outperforms the majority models, except for Hybrid-DRL and Rencos.
For Hybrid-DRL with a better performance on the out-of-domain functions, we ascribe to the advantages of deep reinforcement learning to optimize evaluation metrics.
Compared with Rencos that also considers the retrieved information in the Seq2Seq model, we could ﬁnd that its performance is still lower than HGNN .
On the overall dataset including both of in-domain and out-of-domain data, our model achieves 13.39 , 30.23 and 26.22 , outperforming existing methods by 1.65 , 1.76 and 1.81 in terms of BLEU-4, ROUGE-L and METEOR metrics.
3. A BLATION S TUDY
We also conduct an ablation study to evaluate the impact of different components of our framework, e.g ., retrieval-based augmentation, static graph and dynamic graph in the last row of Table 1.
Consider
HGNN w/o augment where we remove the augmentation component, we found that the performance drops a lot ( e.g ., the BLEU-4 score decreases from 13.39 to 9.87). Furthermore, to investigate the relative impact of retrieved code and summary, we conduct the experiments (see HGNN w/o summary augment and HGNN w/o code augment ).
We found that: 1) Without both code-based and summary-based augmentation (i.e., HGNN w/o augment), the performance drops a lot (compared with HGNN, especially on the in-domain set), which demonstrates the effectiveness of the retrievalbased augmentation.
2) Without either code-based augmentation or summary-based augmentation, we could observe that the performance is also worse than HGNN but is better than removing both of them, demonstrating that both code-based augmentation and summary-based augmentation contribute to the overall performance. Moreover, we found that the summary-based augmentation has more impact than the code-based augmentation. We conjecture that it depends on the speciﬁc task. The summary-based augmentation is more useful because : i) this task is to generate summary and ii) the code and summary are heterogeneous data.
Thus, summary-based augmentation could provide a
[Table]
[Table]
more direct signal for generating better summaries.
However, the code-based augmentation could further improve the performance by enhancing the semantic learning of the program.
Combining them together [Table] our method achieves the best result. Similarly, consider results in HGNN w/o static and HGNN w/o dynamic , we see that: 1) their performance decreases
[Table]
which demonstrates the effectiveness of the Hybrid-GNN and 2) the performance without dynamic graph is worse than the performance without static graph
[Table]
which demonstrates the usefulness of dynamic graph that captures the global structural information.
Similarly, we also evaluate the performance without augmentation and static graph/dynamic graph (see HGNN w/o augment& static and HGNN w/o augment& dynamic ).
Compared with HGNN w/o augment , the results further conﬁrm the effectiveness of the Hybrid-GNN [Table]
3. H UMAN E VALUATION
As shown in Table [Table] we perform a human evaluation on the overall dataset to assess the quality of the generated summaries by our approach, NNGen, Transformer, Rencos and SeqGNN in terms of relevance and similarity.
As depicted in Table 1, NNGen, Rencos and SeqGNN are the best retrieval-based, sequence-based, and graph-based approaches, respectively.
We also compare with
Transformer as it has been widely used in natural language processing.
Inspection on the results, our method can generate better summaries which are more relevant with the source code and more similar with the ground-truth summaries.
3. C ASE S TUDY
To perform qualitative analysis, we present two examples with generated summaries by different methods from the overall data set, shown in Table 3.
We can see that, in the ﬁrst example, our approach learns more code semantics, i.e ., p is a self-deﬁned struct variable.
Thus, we could generate a token object for the variable p .
However, other models can only produce string .
Example 2 is a more difﬁcult function with the functionality to “release reference”, as compared to other baselines, our approach effectively captures the functionality and generates more precise summary. 3. E XTENSION ON THE PUBLIC DATASET
We conducted additional experiments on a public dataset, i.e., the Python Code Summarization
Dataset (PCSD), which was also used in Rencos (the most competitive baseline in our paper).
The total number of code samples in PCSD is 109,726 (Barone & Sennrich, 2017).
This number is comparable to the size (i.e., 95k) of our own CCSD benchmark. We follow the setting of Rencos and split PSCD into the training set, validation set and testing set with fractions of 60%, 20% and 20%. We construct the static graph with the open-source tool (Fernandes et al., 2018). We compare our methods on PCSD against various competitive baselines, i.e., NNGen, CodeNN, Rencos and
Transformer, which are either retrieval-based, generation-based or hybrid methods.
The results are shown in Table 4.
We can see that our method outperforms NNGen, CODENN, Rencos and
[Table]
Transformer by 0.95, 3.27 and 1.12 in terms of BLEU-4, ROUGE-L and METEOR.
We also perform the ablation study on PSCD to demonstrate the usefulness of the static graph (i.e., HGNN w/o dynamic) and dynamic graph (i.e., HGNN w/o static).
The results also demonstrate that both static graph and dynamic graph can contribute to our framework.
In summary, the results on both our released benchmark (C benchmark) and existing benchmark (PCSD) demonstrate the effectiveness of our method.
R ELATED W ORK
Source Code Summarization Early works (Eddy et al., 2013; Haiduc et al., 2010; Wong et al., 2015; 2013) for code summarization focused on using information retrieval to retrieve summaries.
Later works attempted to employ attentional Seq2Seq model on the source code (Iyer et al., 2016) or some variants from code text, i.e., AST (Hu et al., 2018a;
Alon et al., 2018) to generate summaries.
However, these works are based on sequential models, ignoring rich code semantics.
Some latest attempts (LeClair et al., 2020; Fernandes et al., 2018) embedded program semantics into graph neural networks.
However, these works mainly use simple representations, which are limited to learn full program semantics.
Graph Neural Networks Over the past few years, GNNs (Li et al., 2015; Hamilton et al., 2017;
Kipf & Welling, 2016) have attracted increasing attention with many successful applications in computer vision (Norcliffe-Brown et al., 2018), natural language processing (Xu et al., 2018a).
Because by design GNNs can model graph-structured data, recently, some works have extended the widely used Seq2Seq architectures to Graph2Seq architectures for various tasks including machine translation (Beck et al., 2018), and graph (e.g., AMR, SQL)-to-text generation (Zhu et al., 2019; Xu et al., 2018b).
Some works have also attempted to encode programs with graphs for diverse tasks e.g.,
VARNAMING/VARMISUSE (Allamanis et al., 2017), Source Code Vulnerability Detection (Zhou et al., 2019).
As compared to these works, we innovate a hybrid message passing GNN performed on both static graph and dynamic graph for better message fusion.
C ONCLUSION AND F UTURE W ORK
In this paper, we proposed a framework for automatic code summarization.
A novel retrievalaugmented mechanism is proposed for combining the beneﬁts of both retrieval-based and generationbased approaches.
Moreover, to capture global semantics among nodes, we developed a hybrid message passing GNN based on both static and dynamic graphs.
The evaluation shows that our approach improves state-of-the-art techniques substantially.
Future directions include exploring more effective ways to learn graph structures, combining other information, e.g ., API knowledge for code summarization, and graph robustness analysis for GNN.
Appendices A D ETAILS ON C ODE P ROPERTY G RAPH
Code Property Graph (CPG) (Yamaguchi et al., 2014), which is constructed on abstract syntax tree (AST), combines different edges (i.e., “Flow to”, “Control”) to represent the semantics of the program.
We describe each representation combining with Figure 3 as follows:
• Abstract Syntax Tree (AST).
AST contains syntactic information for a program and omits irrelevant details that have no effect on the semantics.
Figure 3 shows the completed AST nodes on the left simple program and each node has a code sequence in the ﬁrst line and type attribute in the second line.
The black arrows represent the child-parent relations among ASTs.
• Control Flow Graph (CFG).
Compared with AST highlighting the syntactic structure, CFG displays statement execution order, i.e., the possible order in which statements may be executed and the conditions that must be met for this to happen.
Each statement in the program is treated as an independent node as well as a designated entry and exit node.
Based on the keywords if , for , goto , break and continue , control ﬂow graphs can be easily built and “Flow to” with green dashed arrows in Figure 3 represents this ﬂow order.
• Program Dependency Graph (PDG).
PDG includes data dependencies and control dependencies : 1) data dependencies are described as the deﬁnition of a variable in a statement reaches the usage of the same variable at another statement.
In Figure 3, the variable “ b ” is deﬁned in the statement “ int b = a++ ” and used in “ call (b) ”.
Hence, there is a “Reach” edge with blue arrows point from “ int b = a++ ” to “ call (b) ”.
Furthermore, Deﬁne/Use edge with orange double arrows denotes the deﬁnition and usage of the variable.
2) different from CFG displaying the execution process of the complete program, control dependencies deﬁne the execution of a statement may be dependent on the value of a predicate, which more focus on the statement.
For instance, the statements “ int b = a++ ” and “ call(b) ” are only performed if a is even.
Therefore, a red double arrow “Control” points from “ if (a % 2) =
= 0 ” to “ int b = a++ ” and “ call(b) ”.
B D ETAILS ON B ASELINE M ETHODS
We compare our approach with existing baselines.
They can be divided into three groups: Retrievalbased approaches, Sequence-based approaches and Graph-based approaches.
For papers that provide the source code, we directly reproduce their methods on CCSD dataset.
Otherwise, we reimplement their approaches with reference to the papers.
B. R ETRIEVAL - BASED APPROACHES
TF-IDF (Haiduc et al., 2010) is the abbreviation of Term Frequency-Inverse Document Frequency, which is adopted in the early code summarization (Haiduc et al., 2010).
It transforms programs into weight vectors by calculating term frequency and inverse document frequency.
We retrieve the summary of the most similar programs by calculating the cosine similarity on the weight vectors.
NNGen (Liu et al., 2018) is a retrieved-based approach to produce commit messages for code changes.
We reproduce such an algorithm on code summarization.
Speciﬁcally, we retrieve the most similar
B. S EQUENCE - BASED APPROACHES
CODE-NN (Iyer et al., 2016; Barone & Sennrich, 2017) adopts an attention-based Seq2Seq model to generate summaries on the source code.
Transformer (Ahmad et al., 2020) adopts the transformer architecture (Vaswani et al., 2017) with self-attention to capture long dependency in the code for source code summrization.
Hybrid-DRL (Wan et al., 2018) is a reinforcement learning-based approach, which incorporates AST and sequential code snippets into a deep reinforcement learning framework and employ evaluation metrics e.g., BLEU as the reward.
Dual Model (Wei et al., 2019) propose a dual training framework by training code summarization and code generation tasks simultaneously to boost each task performance.
Rencos (Zhang et al., 2020) is the retrieval-based Seq2Seq model for code summarization.
it utilized a pretrained Seq2Seq model during the testing phase by computing a joint probability conditioned on both the original source code and retrieved source code for the summary generation.
B. G RAPH - BASED APPROACHES
We also compared with some latest GNN-based works, employing graph neural network for source code summarization.
GCN2Seq, GAT2Seq modify Graph Convolution Network (Kipf & Welling, 2016) and Graph
Attention Network (Velickovic et al., 2018) to perform convolution operation and attention operation on the code property graph for learning and followed by a LSTM to generate summaries.
SeqGNN (Fernandes et al., 2018) combines GGNNs and standard sequence encoders for summarization.
They take the code and relationships between elements of the code as input.
Specially, a
BiLSTM is employed on the code sequence to learn representations and each source code token is modelled as a node in the graph, and employed GGNN for graph-level learning.
Since our node sequences are sub-sequence of source code rather than individual token, we adjust to slice the output of
BiLSTM and concatenate each token representation in node sequences as node initial representation for summarization.
C M ODEL S ETTINGS
We embed the most frequent 40,000 words in the training set with 512-dims and set the hidden size of BiLSTM to 256 and the concatenated state size for both directions is 512.
The dropout is set to 0.3 after word embedding layer and BiLSTM.
We set GNN hops to 3 for the best performance.
The optimizer is selected with Adam with an initial learning rate 0.001.
We also use teacher forcing strategy with forcing probability equals to 0.8 and forcing decay is set to 0.99.
The batch size is set to 64 and early stop for 10.
The beam search width is set to 5 as usual.
All experiments are conducted on the dgx server with four Nvidia Graphics Tesla V100 and each epoch takes averagely 20min.
All hyperparameters are tuned with grid search (Franceschi et al., 2017) on the validation set.
D D ETAILS ON D ATA P REPARATION
It is non-trivial to obtain high-quality datasets for code summarization.
We noticed that despite some previous works (Barone & Sennrich, 2017; Hu et al., 2018b) released their datasets, however, they are all based on high-level programming languages i.e. Java, Python.
Furthermore, they have been conﬁrmed to have extensive duplication to make model overﬁt to the training data that overlapped with the test set (Fernandes et al., 2018; Allamanis, 2019).
We are the ﬁrst to explore summarization on C programming language and make our benchmark public to beneﬁt the community research.
[Table]
->create_rng(lib->crypo,
RNG_STRONG);if(!rng||rng->allocate_bytes(rng,COOKIE2_SIZE,&this->cookie2)){
DESTROY_IF(rng);returnFALSE;} message->add_notify(message,FALSE,COOKIE2,this->cookie2); rng->destroy(rng);returnTrue }
Ground-Truth free a dma descriptor build a cookie and add it to the message NNGen allocate a dma descriptor initialize seeds for spi generation Transformer free a dma descriptor build as10x command header Rencos allocate a dma descriptor initialize seeds for spi generation SeqGNN free the device build a new task HGNN free a dma descriptor build a message cookie
We crawled from popular C repositories (e.g., Linux and Redis) on GitHub, and then extracted separate function-summary pairs from these projects.
Speciﬁcally, we extracted functions and associated comments marked by special characters "/**" and "*/" over the function declaration.
These comments can be considered as explanations of the functions.
We ﬁltered out functions with line exceeding 1000 and any other comments inside the function, and the ﬁrst sentence was selected as summary.
A similar practice can be found in (Jiang et al., 2017).
We totally collected 360k raw function-summary pairs.
Furthermore, functions with token size greater than 150 were removed for computational efﬁciency and there were 130k functions left.
Since duplication is very common in existing datasets (Fernandes et al., 2018), we performed a strict de-duplication process followed by Allamanis (2019) and removed functions with text similarity over 80% and ﬁnally kept 95k+ unique functions.
We name this dataset C Code Summarization Dataset (CCSD).
To testify model generalization ability, we randomly selected some projects as the out-of-domain test set with 2, examples and the remaining were randomly split into train/development/test with 82,656/4,340/4, examples.
The open-source code analysis platform for C Joern (Yamaguchi et al., 2014) was applied to construct code property graphs.
E M ORE EXAMPLES
We show more examples in Table 5 and ﬁnd that HGNN can generate more high-quality summries based on our approch.