Pyramid Attention For Source Code Summarization
Abstract
This paper presents a multi-granularity method for the task of source code summarization, which generates a concise functional description for the given code snippet.
We notice that skilled programmers write and read source codes hierarchically and pay close attention to conceptual entities like statements, tokens, sub-tokens, and the mapping relations between them.
The entities have speciﬁc emphasis according to their granularities, e.g., statements in coarse-granularity reveal the global logical semantics of code, and the sub-tokens in ﬁne-granularity are more related to the textual semantics.
Driven by this observation, we argue that a multi-granularity formulation incorporating these conceptual entities may beneﬁt the code summarization task.
Concretely, the source code is transformed into a pyramidal representation, and then a pyramid attention mechanism is applied for efﬁcient feature aggregation among different hierarchies in it.
We instantiate our multi-granularity method using the proposed pyramid attention and name it PA-former (Pyramid Attention transformer), which is evaluated on two source code summarization benchmarks where it surpasses the prior works and achieves new state-of-the-art results.
Our code and data are available at https://github.com/leichainju/pa-former .
Introduction
Automatic source code summarization is drawing increasing attention as its promising application prospects in software development and maintenance.
This task is challenging due to the complex syntax structure in programs and the arbitrariness of variable naming.
Summarization methods need to exploit textual and grammatical information in programs to learn a meaningful representation for concise description generation.
Following the early stage works [12, 30] that treated the code as a sequence of text and directly adopted the well-developed seq2seq [28] models for this task, recent leading methods [27, 25] further exploit the explicit structure (i e., abstract syntax tree or pair-wise relations extracted from it) in programs to boost the model performance.
Current structure-aware code summarization methods are either in a hybrid way [27, 1, 13, 3] or in a structure-guided way [25, 35, 9, 19].
Works in hybrid fashion encode the AST-based structure using structure-aware models like GGNN [16] or TreeLSTM [29], and then combine the structural representation with the textual representation learned by sequential models for the decoding process.
The structure-guided approaches use the pair-wise relations between tokens (or sub-tokens) extracted from code structure as inductive biases to guide the learning process.
While these methods achieve excellent results, they only model the source code from a single-granularity perspective, which can be further improved in a multi-granularity manner.
To understand a given code snippet, a skilled programmer hierarchically breaks it down into ﬁnegrained entities (i.e., sub-tokens) and builds up conceptual semantics gradually from the ﬁne-grained
[Figure]
Integer
[Figure]
parseInt(
[Figure] [Figure]
[Figure] [Figure]
entities to coarse-grained ones.
We now use the example shown in Figure 1 to illustrate the multigranularity process for code understanding.
Concretely, we ﬁrst segment the whole code into several coarse-grained spans according to punctuations and grammars, where a span corresponds to a statement conceptually (Figure 1(a)).
Given the coarse-grained sequence, we can roughly summarize the code as return something if something is done .
After that, we dive into each statement and do intra-statement segmentation for it (Figure 1(b)).
Now we can detail the ﬁrst something as return something produced by a method invocation .
To further clarify which method is invoked and what is produced , we go deep into each token and further segment token into sub-tokens.
Here, we can ﬁgure out that a int parsing method is invoked and a int is produced (Figure 1(c)).
We ﬁnally roll back the details from ﬁne-grained entities to coarse-grained ones and obtain the overall understanding of the given code.
Driven by the above analysis, we propose our multi-granularity formulation for source code summarization by mimicking human behavior.
We ﬁrst construct a pyramidal input, which is divided into ﬁne-grained layer, semi-grained layer and coarse-grained layer from bottom to top (Figure 1(d)).
And a novel pyramid attention mechanism is proposed for efﬁcient feature aggregation among these layers and produces a pyramidal representation with rich semantics.
Because both coarse-grained logical patterns and ﬁne-grained textual features are considered simultaneously, our model can predict a more comprehensive summary.
Taking the code illustrated in Figure 1(e) as an example, compared with current leading methods, our model notices the try ... catch ... structure in this code snippet and uses the additional if the string can be parsed as an int.
to tell the captured exception handling pattern.
Without bells and whistles, vanilla Transformer [31] equipped with the proposed P yramid
A ttention (PA-former) achieves about 5% performance improvement (evaluated by BLEU [24]) on a Java dataset.
PA-former is evaluated on two source code summarization benchmarks where it surpasses the prior works and reaches new state-of-the-art results.
And ablation studies are conducted to show the efﬁciency of the proposed method.
Related Work
Source code representation.
Learning-based source code comprehension is drawing increasing attention as its promising application prospects in the ﬁeld of software development and software maintenance.
A meaningful representation can be used for various downstream tasks, such as code clone detection [34, 33], code classiﬁcation [23, 37,
[7], etc.
Early learning-based works treated source code as a sequence of text ignoring the structure features.
Mainstream base models like CNN [2], Attention-based LSTM [12] and Transformer [30] were all used directly to capture the textual patterns in programs.
Many follow-ups leveraged both textual and structure information to learn a more meaningful code representation.
Tree-based models like TBCNN [23], Code-GRU [17], ASTNN [37] were proposed to represent the source code by considering the tree structure.
Some works [1] extended ASTs into graphs by introducing control ﬂows and data ﬂows, then a GGNN was applied to represent the code graphs.
The set of paths [3, 4] extracted from AST was also used as a representation of the corresponding code.
Some works [11, 8, 10] ﬂatten the ASTs into a sequence, and then the off-the-shelf sequential models are applied to learn the textual and syntax features.
Currently, most attention is focused on guiding the attention computation in Transformer by using the structure bias in programs [25, 9, 38].
Source code summarization.
To generate a functional description for the given code snippet, code summarization models should capture both textual and structural information in programs.
Some early works [12, 30, 32] just treated the source code as sequence or ﬂatten the AST using a structurebased traversal (SBT) method [10] and then performed a conventional seq2seq learning.
The hybrid works represented both textual and structural features separately, then a hybrid mechanism is used to combine the learned representations for summary generation.
HDeepcom [11] applied the SBT method for structural representation and CAST [27] learning the structure via hierarchical splitting and reconstruction of abstract syntax trees.
Many Transformer-based methods [27, 1, 13, 3] used structure information to guide the attention computation.
TPTrans [25] encoded the pairwise path between code tokens and incorporated the path embeddings like relative position encoding.
Unlike the above-mentioned methods which view the source code as a single-granularity data, we propose a multi-granularity source code summarization method.
Multi-scale feature learning.
The multi-scale feature
processing is widely used in the computer vision community.
FPN [21] leverages the pyramid-shaped features of ConvNet and creates a feature pyramid that has rich semantics at all scales.
Recently, the multi-scale feature hierarchies were also introduced into Vision Transformer (ViT) by a variety of works like MViT [6], Swin [22] and
HRFormer [36], which show strong evidence that Transformer could work well with the multi-scale features.
Approach
As shown in Figure 2, given a source code, we ﬁrst construct sequences with three granularities and form them into a pyramid-shaped structure based on the mapping constraints between different granularities.
After that, the proposed pyramid attention mechanism (in Figure 3) is applied layer by layer in the encoding stage to produce a feature pyramid which is used by a vanilla Transformer decoder to generate the summary.
We introduce the details of the proposed pyramid input constructor, pyramid attention mechanism and implementation details in this section.
3.1 Pyramidal Input Constructor
We construct the pyramidal input following the human code comprehension process described in
Section 1.
Entities in different granularities have their own emphases, telling speciﬁc attributes of source code from different viewpoints.
For instance, the coarse-grained statement sequence reveals the global logical semantics of code, the semi-grained token sequence shows the code grammar details, while the ﬁne-grained sub-token split from each token is attached with textual information.
We introduce the details of the proposed pyramid input constructor in the following.
Fine-grained sequence.
Each leaf node in the AST corresponds to a token (like boolean , value ) in the source code.
Since method names and variable names in programs are usually composed by multiple natural language words with CamelCase or snake_case , we can further split these combined tokens into sub-tokens, which reveal the textual information of source code.
To construct the ﬁnegrained layer (i.e., sub-token sequence), we ﬁrst obtain the leaf nodes sequence of AST by depth-ﬁrst pre-order traversal and then split each token into sub-tokens.
Such a sub-token technique is widely used by source code summarization methods [30, 25, 27] and proved to be effective.
Formally, let
[Figure]
Output
[Figure]
T =
[ t 1 , t 2 , . . . , t N ] ( T for text) denote the ordered sub-token sequence, that is, the ﬁne-grained input representation situated in bottom of the pyramidal input.
For the example in Figure 1, T is [public, static, int, parse, String, As, Int, (, ...] .
Semi-grained sequence.
We use the attribute of the parent node of each token (i.e. leaf) in parsed
AST to represent the token’s grammatical property.
The semi-grained sequence is constructed by replacing the element in ordered token sequence with its grammatical property directly.
Formally, let G =
[ g 1 , g 2 , . . .
, g M ] ( G for grammar) denote the ordered token grammatical sequence, that is, the semi-grained input representation situated in middle of the pyramidal input.
For the example in Figure 1, G is [modifier, modifier, type, identifier, (, ...] .
In addition, there is a one-to-many mapping relationship between token and sub-token.
We denote it as M G→T , M
G→T ( g i , t j ) = 1 if the sub-token t j is split from g i , which will be used to guide the information aggregation process from ﬁne-grained hierarchy to semi-grained hierarchy.
Coarse-grained sequence.
We use the nodes corresponding to statements to form our coarsegrained sequence, which reveals the logic information of source code.
As the statement type is limited, we directly identify the AST node as statement-level node if the tag belongs to the pre-deﬁned statement type set (see Appendix C for more details).
Formally, let L =
[ l 1 , l 2 , . . .
, l K ] ( L for logic) denote the ordered statements sequence, that is, the coarse-grained input representation situated in top of the pyramidal input.
For the code in Figure 1, L is [declaration, parameters, try, return_statement, ...] .
Similar to the semi-grained sequence, we also construct a one-to-many mapping function M L→G between statement and token, which bridge the coarse-grained hierarchy with the semi-grained hierarchy.
M L→G ( l i ,g j ) =
1 if the statement l i is the most recent ancestor of token g j among L in AST.
In summary, we construct a sequence set {T , G , L} in three granularities which helps the model to capture patterns from different scales and extract mapping relationships {M L→G , M G→T } between adjacent granularities which introduce inductive bias to aid the model to learn accurate semantics.
Thus, the sequences and the mapping relationships all together form the pyramidal input representation of the given code.
3.2 Pyramid Attention
The pyramidal layers {T , G , L} complement each other following the mapping relationships {M
L→G , M G→T } .
We propose a novel pyramid attention mechanism for efﬁcient feature aggrega-
[Table]
[Table] [Table]
/42bNgdtfTDweG+GmXl+wqhUtv1tVFZW19Y3qpu1re2d3T1z/6An41QQ6JKYxaLvYwmMcugqqhj0EwE48hnc+5Obwr9/ACFpzDtqmoAX4RGnISVYaWloHrmPNABFWQCZG2E1JphlnTwfmnW7Yc9gLROnJHVUoj00v9wgJmkEXBGGpRw4dqK8DAtFCYO85qYSEkwmeAQDTTmOQHrZ7IHcOtVKYIWx0MWVNVN/T2Q4knIa+bqzuFEueoX4nzdIVXjlZZQnqQJO5ovClFkqtoo0rIAKIIpNNcFEUH2rRcZYYKJ0ZjUdgrP48jLpnTecZuPirllvXZdxVNExOkFnyEGXqIVuURt1EUE5ekav6M14Ml6Md+Nj3loxyplD9AfG5w/BxJcq</latexit>
G <latexit sha1_base64="w7GkJ7R/PBAv+q4CWvfdiGrNPwQ=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBIvgqsxIRZdFF7qsYB8wHUomzbShmWRIMkIZ+hluXCji1q9x59+YaWehrQcCh3PuJeeeMOFMG9f9dkpr6xubW+Xtys7u3v5B9fCoo2WqCG0TyaXqhVhTzgRtG2Y47SWK4jjktBtObnO/+0SVZlI8mmlCgxiPBIsYwcZKfj/GZkwwz+5mg2rNrbtzoFXiFaQGBVqD6ld/KEkaU2EIx1r7npuYIMPKMMLprNJPNU0wmeAR9S0VOKY6yOaRZ+jMKkMUSWWfMGiu/t7IcKz1NA7tZB5RL3u5+J/npya6DjImktRQQRYfRSlHRqL8fjRkihLDp5ZgopjNisgYK0yMbaliS/CWT14lnYu616hfPjRqzZuijjKcwCmcgwdX0IR7aEEbCEh4hld4c4zz4rw7H4vRklPsHMMfOJ8/fASRZQ==</latexit> input, and outputs a multi-scale representation without changing the original pyramid shape.
The whole process consists of a bottom-to-top aggregation pathway and a cross-granularity information interaction process.
tion among these layers.
As illustrated in Figure 3, pyramid attention module includes a bottom-totop aggregation pathway and a cross-granularity information interaction process.
We describe the details in the following.
Bottom-to-top aggregation.
The bottom-to-top pathway does a ﬁne-to-coarse information aggregation, which ensures to enrich the higher level representation using the lower level semantics.
Given the pyramid input data: {T , G , L} and the mapping function {M L→G , M G→T } , we aggregate the representation of T into G w.r.t.
M G→T .
The formulation of the aggregation process is given as:
[Equation]
Where W agg ∈ R D × D , b agg ∈ R D are learnable parameters, g i , t
i ∈
R D are the i -th representation vectors in G and T respectively, D represents the dimension of the representation, and b g i is the aggregated representation of g i .
The aggregation process from G into L w.r.t.
M L→G is the same as above.
We further conduct a aggregation directly from T into L w.r.t.
M L→T , which is obtained by:
[Equation]
In summary, we update {G , L} into { b G , b L} using T and {T , G} respectively.
The mapping-based aggregation is reasonable and necessary as the semantics of the element in high levels come from the combination of its descendants’ semantics.
The semi-grained sequence and coarse-grained sequence are initialized as the grammatical type of the tokens and statements.
Figuratively speaking, they are like torsos with empty skeletons, and we need to ﬁll them with the textual information to make them complete.
There is a ablation experiment demonstrating its effectiveness in Section 4.3.
Cross-granularity interaction.
The aggregated semantics of the elements in semi/coarse-grained sequences tend to be local.
To alleviate this dilemma, we further make the elements in a speciﬁc granularity to query all elements at three granularities, which guarantees that the model can capture both ﬁne-grained and coarse-grained features.
Formally, we now have {T , b G , b L} after aggregation.
Taking sequence b G ∈ R M × D of length M in certain granularity as an example, G will query all of {T , b G , b L} in the cross-granularity interaction process.
We ﬁrst concatenate the pyramid representation as
[Equation]
here N, M, K are the sequence lengths of T , b G , b L respectively.
A relative position encoded multihead attention [26] is applied here and we use the relative position map R b G ∈ Z M ×|C| to make the model granular-aware.
The multi-head cross-attention (MHCA) process is calculated as:
[Equation]
where W hq , W hk , W hv ∈ R DH × DH , h ∈ [1 , . . .
, H ] are projection matrices of Query, Key and Value for h -th head.
W o ∈ R D
× D is projection matrix for attention output.
H represents the number of heads and b X represents the output of multi-head self-attention process.
W b G rel _ k , W b G rel _ k ∈ R DH × DH are projection matrices of Key and Value of relative position embedddings, the b G means that thay are speciﬁc to b G .
[Table]
[Table] [Table] [Table]
[Table] [Table]
L <latexit sha1_base64="zvOHGED9Rb/HhQfIezQ4uA
[Table]
/CWT14lnYu616hfPjRqzZuijjKcwCmcgwdX0IR7aEEbCEh4hld4c4zz4rw7H4vRklPsHMMfOJ8/fASRZQ==</latexit>
For the construction of relative position map R , we calculate the clipped relative distance with max distance as 32 for the pair within granularity and assign the relative distance as a default value (e.g. max distance + 1 and max distance + 2) for all the pair across the speciﬁc granularity.
Figureshows a toy example of granular-aware relative position map with max distance as 3 for G crossgranularity interaction.
The cross-granularity interaction for T and L is similar to b G .
We share attention parameters (i.e. W hq , W hk , W hv and W o ) across three granularities,but assign speciﬁc relative path parameters for each granularity.
In summary, in this cross-granularity interaction process, semantics of the feature pyramid are updated as:
[Equation]
3.3 Pyramid Attention Transformer
As illustrated in Figure 2, the overall model is similar to the vanilla Transformer.
The self-attention module in encoder is replaced with the proposed pyramid attention module for an efﬁcient process of the multi-scale input.
Notably, we only use the ﬁne-grained sequence in the pyramidal representation for decoding, which already contains enough semantics for summary generation and reduces the complexity of the model design.
Following prior works [30], we also apply a copy mechanism in our model which reduces the risk caused by out-of-vocabulary and speeds up training convergence.
Although there is overlap between the vocabularies of different granularity, the meanings of these common words are quite different when differentiated at different granularities.
For instance, else maybe just a natural word in the textual sequence, but a keyword for control in a grammatical sequence.
Thus, we assign individual embedding layer for each granularity to avoid such confusion.
For efﬁcient parameter usage, a parameter sharing strategy is also applied in the cross-granularity interaction process as stated in Section 3.2, We ﬁnd the self-attention mechanism is strong enough to handle this multi-role mission.
Thus, our model doesn’t add too many parameters compared to
Transformer.
What’s more, such a parameter sharing strategy also brings efﬁcient model implementation.
We only need to concatenate these three sequences and the relative position maps separately, then directly adopt a relative position encoded multi-head self-attention layer to implement the crossgranularity interaction module.
Experiments 4.1 Experiments Setup
Dataset and Metrics.
To demonstrate the effectiveness of the proposed method, we conduct experiments on two widely-used and well-developed java datasets: EMSE-DeepCom 2 [11] which is collected from GitHubs Java repositories and FunCom 3 [14] which has ∼ 2 million java methodcomment pairs.
We ﬁlter out the examples that cannot be parsed properly or are of too large size for both datasets.
And deduplication is done on EMSE-DeepCom to avoid data leakage.
Table 1 shows the statistics of the datasets.
[Table]
We use 6 metrics to evaluate the model performance on the datasets.
BLEU [24], Rouge-L [20] and Meteor [5] are widely used to evaluate the quality of text similarity in the domain of natural language generation.
Another 3 metrics are used to qualify the token-wise prediction performance.
We use beam search with a beam size of 3 for all models in inference phase.
Data preprocessing.
All data preprocessing of code is based on the parsed syntax tree.
We obtain the AST using the open-sourced tool Tree-sitter 4 .
Please refer to 3.1 for details of the proposed pyramid data construction.
We re-implement the data preprocessing pipeline for all compared baselines based on the AST parsed by Tree-sitter, please refer to Appendix C for more details.
Baselines.
Early works viewed source code as a sequence of text and used the sequential model like LSTM or Transformer, we mainly compare with CODE-NN [12] and NeuralCodeSum [30] for this type of methods.
A few works only learned the structure features for this generation task, we compare with TreeLSTM [29] here.
Most works both exploited the textual and structural information for this task.
One group of works learned to represent these two types of information separately, then a hybrid mechanism is used to combine them in the decoding phase.
We do comparison with
HDeepCom [11], ASTAttnGRU [13] and CAST [27] for the hybrid fashion.
Another line of methods used structure relations as an inductive bias to guide the information interactions in the model.
Here, we compare with SiT [35], GREAT [9] and TPTrans [25].
For clarity, the backbone and taxonomy of each method are listed in Table 2.
And we add a copy mechanism for all transformer-based models.
We directly adopt the model implementation script if the code is publicly available, and re-implement the method according to the description in the corresponding paper if the code is not provided or not available.
What’s more, all baselines are integrated under a uniﬁed PyTorch-based code summarization framework developed by ourselves for fair comparisons.
We provide implement details for baselines in Appendix A.3.
Hyperparameters and training setup.
For fair comparisons, all the Transformer-based models use the default Transformer conﬁgurations with embedding dimension as 512, feedforward dimension as 2048, head number as 8, and layer number for encoder/decoder as 6 and all RNN-based models use the hidden dimension with 512. For embeddings, we use the embedding dimension with
http:/ /https://
[Table]
512 for all approaches.
The size of vocabularies is limited by 50000 over code sequences andover summary sequences in EMSE-DeepCom, and the corresponding conﬁgurations are 35000 and 30000 in FunCom.
All model is trained using NVIDIA Tesla A100 GPUs with a batch size of 64.
We train all baselines including our models using AdamW optimizer with a multi_step learning rate scheduler, and set the initial learning rate to 0.0002 and 0.003 for Transformer-based and RNN-based models, respectively.
And warmup strategy is used for stable training in the Transformer-based model.
The training epochs for each method are listed in Table 2.
4.2 Main Results
Comparisons with baselines.
The comparisons with baselines are listed in Table 2.
Without bells and whistles, we ﬁnd that our proposed PA-former substantially surpasses all baselines and achieves new state-of-the-art on both middle-size and large-size datasets.
Compared with NeuralCodeSum which is a relative positional encoded Transformer, our method has a 5% improvement in BLEU on the middle-size dataset.
For the current leading methods CAST and TPTrans, which both take all the nodes in ASTs and the sub-token sequence as input, our model only takes a subset of AST nodes and sub-tokens surpass both of them over 1.0+ BLEU scores on two datasets.
And thanks to the efﬁcient implementation and parameter sharing strategy, our model requires far less training time to converge, while TPTrans needs a very long time to converge as the large size pair-wise paths and the sparse scatter algorithm.
The leading results demonstrate that our multi-scale formulation for the task of source code summarization is reasonable.
To further evaluate the effectiveness of our method, we provide a case study in Appendix D. Moreover, we provide the results on RMSE-Deepcom dataset with error bars to show that the improvement achieved by our method is statistically signiﬁcant in
Table 11 at Appendix B.
Human evaluation.
As a complement, we conduct human evaluation over the transformer-based methods following the previous works [12, 27, 15].
We invite 5 volunteers with over 5 years of programming experience and ask them to score (0 - 10, 10 is the best) the generated summaries over 100 examples sampled randomly from the testing sets (20 from RMSE-Deepcom and 80 from Funcom).
Here we focus on two key points: naturalness (grammaticality and ﬂuency of the generated summary) and usefulness (what extent the generated summary is useful to understand the code).
We divide the 100 examples into 5 groups (each one has 20 examples) and assign each group intodifferent volunteers, and the ﬁnal score of each example is the average of its 3 "review" scores.
As shown in Table 3, our PA-former outperforms others in both naturalness and usefulness. [Table]
4.3 Ablation Study
In this part, we analyze the importance of designs in our proposed PA-former using a series of ablation studies on EMSE-DeepCom dataset.
Unless otherwise speciﬁed, the training setup is the same as above.
Multi-scale v.s. single-scale.
To demonstrate the effectiveness of multi-granularity formulation, we degenerate the pyramid input into a single large sequence by concatenating the three granularity sequences and directly feed the same sequence into a relative positional encoded Transformer with a copy mechanism.
As shown in Tabel 4, more data in the input sequence obtains performance improvement compared to NeuralCodeSum which only takes a sub-token sequence as input.
However, processing the same data using our proposed multi-scale way achieves higher performance.
[Table]
Effectiveness of pyramid attention.
The core components of pyramid attention are the bottomto-top aggregation and cross-granularity interaction.
We show the necessity for both of these two components here.
To demonstrate the effectiveness of bottom-to-top aggregation, we evaluate a variant of PA-former whose bottom-to-top pathway is disabled.
As shown in Table 5, the model performance drops by about 1.1 BLEU scores without such a component.
To demonstrate the effectiveness of cross-granularity interaction, we evaluate a variant of PA-former whose cross-granularity interaction is disabled which directly degenerates into NeuralCodeSum as we only use the ﬁne-grained encoder output for decoding.
As shown in Table 5, the model performance drops by about 1.7 BLEU scores without such a component.
Such results reveal the importance of cross-granularities interaction.
We further disable the granular-aware relative embedding to investigate its importance, and we observe a big drop in performance shown in Table 5, which shows its effectiveness.
[Table]
Decoding strategy.
The output of our PA-former encoder is a pyramid representation, thus there are various ways for decoding.
Our PA-former directly uses the ﬁne-grained sequence just for simplicity.
Here, we study other implementations that utilize all three granularity of representation.
The most intuitive method is concating the three granularity representations [Table] and another way is using them in a serial strategy [18].
As Table 6 shows [Table] using the ﬁne-grained representation is good enough in this task to beneﬁt from the design of information interaction across granularities.
The results are a bit counterintuitive, we think a reasonable explanation is that the ﬁne-grained representation already contains enough information beneﬁted from the proposed pyramid attention.
When all three granularities are used, there is severe information redundancy between different granularities, which complicates the task.
[Table]
Grammar input.
When constructing the pyramid input, we intentionally enforce the bias between different granularities by assigning different types of content.
For example, we bias the semi-grained sequence into grammar features by assigning token grammar types to it.
To evaluate the rationality of assigning different types of content, we compare the proposed pyramid input without grammar information, which set the initial content of semi-grained and coarse-grained sequences to empty.
The results in Table 7 prove the rationality of pyramid input design.
[Table]
Conclusion
In this paper, we present a multi-scale formulation for source code summarization.
Given a code snippet, we construct a pyramidal input with three granularities.
And a novel pyramid attention mechanism is proposed for efﬁcient information interaction across granularities.
Without bells and whistles, the proposed method obtains 5% improvement over the strong baseline and achieves new state-of-the-art results.
Additional ablation studies show the effectiveness of the proposed modules.
Furthermore, our multi-scale formulation for the source code can be easily extended to other codeunderstanding tasks.
In future work, we would try to scale our model up and pre-train it on larger source code corpora to further improve its capability and generalizability on source code comprehension tasks.
Acknowledgement
This research was supported by NSFC (62076121, 61921006).
The authors would like to thank Ying
Li, Hao-Yuan He, Ren-Biao Liu, Yu-Xi Sun and Zhi-Yu Shen for helpful discussions.
Finally, We are grateful to all reviewers for their insightful comments.
Appendix
We ﬁrst provide the implementation details of PA-former and the compared baselines (Appendix A).
Then, more supplementary experiments will be listed in Appendix B. Next, we introduce the data process for code (Appendix C).
Finally, qualitative examples are provided in Appendix D.
A Implementation Details A.1 PA-former
Pyramid attention
Here we provide the pseudocode of our pyramid attention forward process in a PyTorch-like style in Algorithm 1.
Compared with other methods, the algorithm implementation is straightforward and efﬁcient.
Please refer to the provided code in pa_former for more details.
[Table]
A.2 Baselines
In this section, we provide the implementation details of baselines.
CODE-NN [12] is a basic attention-based seq2seq model which takes a token sequence as input.
For the encoder, we use a 2-layer bi-directional LSTM and set the hidden size to 256.
For decoder, we use a 2-layer single-directional LSTM and set hidden size to 512.
Following [30], the hidden and cell vectors of the bi-directional encoder is concated separately to form 512-D hidden vectors which are used as the initial hidden states for decoder.
And the copy mechanism is used over the encoder-decoder framework.
This method directly uses the token sequence, which leads to a big risk of out-of-vocabulary and limits the performance of this method.
TreeLSTM [29] is a attention-based tree2seq model which takes a parsed AST as input.
For the encoder, we use child-sum tree-based LSTM with the hidden size of 512 and implement it using the excellent Deep Graph Library (DGL) 5 .
For the decoder, we use a 2-layer single-directional LSTM and set the hidden size to 512.
The hidden and cell vectors of the root node are used as the initial hidden states for decoder.
And a copy mechanism is used over the encoder-decoder framework.
As the large size of AST and the risk of out-of-vocabulary, such a structure-aware doesn’t obtain large performance improvement.
HDeepCom [11] is a hybrid method which encoder the ﬂattened AST sequence and sub-token sequence separately, and combines these two representation when decoding using the attention mechanism.
We set the embedding size and hidden size to 512 for fair comparison.
ASTAttnGRU [13] is also a hybrid method that encodes the AST-based graph and sub-token sequence separately, and combines these two representations when decoding using the attention mechanism.
We use the DGL to implement the tree encoder and set the embedding size and hidden size to 512 for a fair comparison.
SiT [35] uses the structure information (abstract syntax edges, control ﬂow edges and data dependency edges) as attention mask to guide the attention computations.
Following the paper, this method only considers the punctuation-removed token sequence.
Copy mechanism is applied and the model conﬁguration is consistent with the default Transformer.
GREAT [9] introduces various edge types following [1] and models the pairwise relations using the summed edge type embeddings between two terminal nodes (i.e., token) in AST.
Then these relations are used as relative distance bias in the attention computation process as Equation 4.
They only consider the token sequence and embed the tokens by averaging embeddings of its sub-token(s).
We also add a copy mechanism over the encoder-decoder framework for a fair comparison.
NeuralCodeSum [30] uses the sub-token sequence as input and directly adapts relative position encoded Transformer.
And copy mechanism is also introduced in this method.
CAST [27] is a hybrid method that encodes the sub-token sequence using the Transformer encoder and split the AST into statement subtrees for structure representation learning.
Then use tree-based
RvNN to encode each subtree and then a max-pooling is applied to get the statement-level structure representations.
Next, these statement-level representations are combined into a tiny tree according to the AST to capture global structure information.
For the decoder, a serial attention mechanism is used to combine the learned sequential and structural features.
And copy mechanism is also used in this method.
TPTrans [25] integrates the absolute paths and relative paths between sub-tokens into Transformer by representing the paths as order non-terminal node sequences and encoding them using GRUs.
The learned path embeddings are used as a relative relation bias in attention computations as Equation 4.
And copy mechanism is also used in this method.
A.3 Training conﬁguration
We show the details of training conﬁgurations in Table 8.
The "rnn-24e" is used for all LSTM/GRUbased methods on both middle-size and large-size datasets.
The "Transformer-30e" is used for all Transformer-based methods on middle-size dataset.
The "Transformer-36e" is used for all
Transformer-based methods on large-size dataset.
For the items of "lr scheduler", "MS-(12, 19)" means that MultiStep learning rate scheduler which update the current_lr into 0 .
1 × current_lr at 12th and 19th epoch.
[Table]
B More Experiment Results
Improvement for Transformer.
we show the effect of training tricks on improving model performance in Table 9.
The results shows the optimizer AdamW and warmup strategy really helps.
Such a improvement caused by training tricks also shows the strong ability of Transformer.
And we use this training settings to all compared baselines for fair comparison.
[Table]
Number of encoder layer.
We further perform the ablation study by varying the number of encoder layers and list the results in Table 10.
As the results show, a deeper model helps improve performance as more cross-granularity aggregations and interactions bring richer semantic information.
[Table]
Results with error bar.
We train the models 3 times with different random seeds on RMSEDeepcom dataset and list the scores with error bars in Table 11.
The results suggest that our method achieves signiﬁcant improvement (e
[Table]
g. 1.5 on BLEU) over the baselines with low standard deviation ( < 0 . 1 ).
[Table]
C Data preprocessing
For the summaries of code which are natural language sentences, we preprocess them following the natural language process (NLP) pipeline.
In this section, we mainly focus on the code preprocessing pipelines which are based on the parsed AST.
Given a code snippet, we use Tree-sitter to get the parsed AST, an example is illustrated in Figure 5.
C.1 Pyramid input
We show the details of our proposed pyramid input construction with the example in Figure 5.
The ﬁne-grained sequence T is easy to handle.
We ﬁrst extract the ordered terminal nodes from
AST, we get [public, int, parseStringAsInt, (, String, s, ), ...] for the given example.
And then we split each element in the token sequence into sub-tokens and we get T
= [public, int, parse, String, As, Int, (, String, ...] .
Here, we ignore the grammatical information and just treat elements of T as natural words, so we name it as T ext sequence.
We use the semi-grained sequence G to represent the G rammatical details of the code, which is a complement to T .
Each element in G corresponds to an element in the token sequence.
For
// parsers a string as a int.
public int parseStringAsInt(String s){ return Integer.parseInt(s); }
A Java Code Snippet method_declaration modifier int_type identifier formal_parameters public int parseStringAsInt ( ) formal_parameter
String in type_identifier identifier block { } return_statement return ; method_invocation identifier identifier argment_list .
Integer parseInt ) ( statement node non-terminal node terminal node in each token, its parent node in AST represents its grammatical property, e.g., parseStringAsInt is an identifier and public is a modifier .
However, we notice that all variable names and method names are marked as identifier .
To further differentiate them, we instead use their grandparents’ node in ASTs to mark the grammatical types, e.g. parseStringAsInt is marked as method_declaration and parseInt is marked as method_invocation .
Finally, we get G =
[modifiers, int_type, method_declaration, (, type_identifier, ...] .
For the mapping relations M G→T ∈ R |G|×|T | , we set M G→T
( i, j ) to 1 if T [ j ] is split from the token corresponding to G [ i ] .
In this example, we set M G→T (3 , 3) , M G→T (3 , 4) , M G→T (3 , 5) and M
G→T (3 , 6) to 1, as [parse, String, As, Int] is split from parseStringAsInt .
The L ogical nature of the code is mostly in statement-level coarse-grained sequence L .
As the type of statement is limited, we ﬁrst set the pre-deﬁned statement set S shown in Table 12 for java dataset.
In addition to the normal java statements, we also treat method_declaration and formal_parameters as statements.
We treat the node in AST as statement node, if its tag is in S .
Thus we get L =
[method_declaration, formal_parameters, return_statement] (yellow nodes in Figure 5).
For the mapping relations M L→G ∈ R |L|×|G| , we set M L→G ( i, j ) to 1 if L
[ i ] is the nearest ancester of G
[ j ] in the scope of L .
In this example, we set M
L→G (2 , 4) , M L→G (2 , 5) , M L→G (2 , 6) and M L→G (2 , 7) to 1, as formal_parameters is the nearest ancester of (, String, in and ) .
[Table]
C.2 Data preprocessing for baselines
Since some compared baselines don’t provide available preprocessing code, we have to implement the preprocessing pipelines according to their papers.
For GREAT [9], we add the additional edges to AST according to construct the code graph representation according to [1].
The edges include NextToken, LastRead, LastWrite, ComputeFrom
LexicalUse and LeafCFG .
The edge LeafCFG represents the edge of control ﬂow graph (CFG) (see details in [9]) and please refer to [1] for details about other edges.
For SiT [35], we add Flow edge between each in-statement token pair, add adopt LexicalUse ,
LastRead and LastWrite in GREAT into the Data dependency edge.
D Qualitative Examples
In this section, we provide several samples of code summarization with different models.
We can see that PA-former generates most precise comment in most cases.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
PA-former (ours): checks if a point falls within a pie chart.
TPTrans: determines if a point is on a pie chart.
CAST: determines if a point is on the pie chart.
NeuralCodeSum: returns true if the speciﬁed point is on the pie chart.
GREAT: checks if point is within pie pie pie pie pie pie pie area.
SiT: checks if a point is on the chart.
ASTAttnGRU: checks if at the current position.
[Table]
[Table] [Table]
this.
[Table]
title;
[Table]
Reference: create xmlreporter which writes to the given stream.
PA-former (ours): create an xml reporter which writes to the given stream.
TPTrans: creates a new xml reporter.
CAST: create an xml reporter to write to the given stream.
NeuralCodeSum: create an instance of this class.
GREAT: create an xml stream.
SiT: create a new reporter.
ASTAttnGRU: create an object decoding the values from the passed stream.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
form
[Table]
unk
[Table] [Table]
a
[Table]
box.
ASTAttnGRU:
[Table]
a
[Table]
box.