Pyramid Attention For Source Code Summarization
Abstract
This paper presents a multi-granularity method for source code summarization, which generates a concise functional description for the given code snippet.
We notice that skilled programmers write and read source codes hierarchically and pay close attention to conceptual entities like statements, tokens, sub-tokens, and the mapping relations between them.
The entities have speciﬁc emphasis according to their granularities, e.g., statements in coarse-granularity reveal the global logical semantics of code, and the sub-tokens in ﬁne-granularity are more related to the textual semantics.
Driven by this observation, we demonstrate that a multi-granularity formulation incorporating these conceptual entities beneﬁt the code summarization task.
Concretely, the source code is transformed into a pyramidal representation, and then a pyramid attention mechanism is applied for efﬁcient feature aggregation among different hierarchies in it.
We instantiate our multi-granularity method using the proposed pyramid attention and name it PA-former (Pyramid Attention transformer). We evaluated it on two source code summarization benchmarks where it surpasses the prior works and achieves new state-of-the-art results.
Our code and data are available at https://github.com/leichainju/pa-former .
Introduction
Automatic source code summarization is drawing increasing attention to its promising application prospects in software development and maintenance.
This task is challenging due to the complex syntax structure in programs and the arbitrariness of variable naming.
Summarization methods need to take advantage of textual and grammatical information in programs to learn a meaningful representation for concise description generation.
Following the early stage works [12, 30] that treated the code as a sequence of text and directly adopted the well-developed seq2seq [28] models for this task, recent leading methods [27, 25] further exploit the explicit structure (i e., abstract syntax tree or pair-wise relations extracted from it) in programs to boost the model performance.
Current structure-based code summarization methods are either in a hybrid way [27, 1, 13, 3] or in a structure-guided way [25, 35, 9, 19].
Works in hybrid fashion encode the AST-based structure using structure-aware models like GGNN [16] or TreeLSTM [29], and then combine the structural representation with the textual representation learned by sequential models for the decoding process.
The structure-guided approaches use the pair-wise relations between tokens (or sub-tokens) extracted from code structure as inductive biases to guide the learning process.
While these methods achieve excellent results, they only model the source code from a single-granularity perspective, which can be further improved in a multi-granularity manner.
To understand a given code snippet, a skilled programmer hierarchically breaks it down into ﬁnegrained entities (i.e., sub-tokens) and builds up conceptual semantics gradually from ﬁne-grained
[Figure]
[Figure]
entities to coarse-grained ones.
We now use the example shown in Figure 1 to illustrate the multigranularity process for code understanding.
Concretely, we ﬁrst segment the whole code into several coarse-grained spans according to punctuations and grammar, where a span corresponds to a statement conceptually (Figure 1(a)).
Given the coarse-grained sequence, we can roughly summarize the code as return something if something is done .
After that, we dive into each statement and do intra-statement segmentation for it (Figure 1(b)).
Now we can detail the ﬁrst something as return something produced by a method invocation .
To further clarify which method is invoked and what is produced , we go deep into each token and further segment the token into sub-tokens.
Here, we can ﬁgure out that an int parsing method is invoked and an int is produced (Figure 1(c)).
We ﬁnally roll back the details from ﬁne-grained entities to coarse-grained ones and obtain the overall understanding of the given code.
Driven by the above analysis, we propose our multi-granularity formulation for source code summarization by mimicking human behavior.
We ﬁrst construct a pyramidal input, which is divided into ﬁne-grained layer, semi-grained layer and coarse-grained layer from bottom to top (Figure 1(d)).
And a novel pyramid attention mechanism is proposed for efﬁcient feature aggregation among these layers and produces a pyramidal representation with rich semantics.
Because both coarse-grained logical patterns and ﬁne-grained textual features are considered simultaneously, our model can predict a more comprehensive summary.
Taking the code illustrated in Figure 1(e) as an example, compared with current leading methods, our model notices the try ... catch ... structure in this code snippet and uses the additional if the string can be parsed as an int.
to tell the captured exception handling pattern.
Without bells and whistles, vanilla Transformer [31] equipped with the proposed P yramid
A ttention (PA-former) achieves about 5% performance improvement (evaluated by BLEU [24]) on a Java dataset.
PA-former is evaluated on two source code summarization benchmarks where it surpasses the prior works and reaches new state-of-the-art results.
And ablation studies are conducted to show the efﬁciency of the proposed method.
Related Work
Source code representation.
Learning-based source code comprehension is drawing increasing attention as its promising application prospects in the ﬁeld of software development and software maintenance.
A meaningful representation can be used for various downstream tasks, such as code clone detection [34, 33], code classiﬁcation [23, 37,
[7], etc.
Early learning-based works treated source code as a sequence of text ignoring the structure features.
Mainstream base models like CNN [2], Attention-based LSTM [12] and Transformer [30] were all used directly to capture the textual patterns in programs.
Many follow-ups leveraged both textual and structural information to learn a more meaningful code representation.
Tree-based models like TBCNN [23], Code-GRU [17], ASTNN [37] were proposed to represent the source code by considering the tree structure.
Some works [1] extended ASTs into graphs by introducing control ﬂows and data ﬂows, then a GGNN was applied to represent the code graphs.
The set of paths [3, 4] extracted from AST was also used as a representation of the corresponding code.
Some works [11, 8, 10] ﬂatten the ASTs into a sequence, and then the off-the-shelf sequential models are applied to learn the textual and syntax features.
Currently, most attention is focused on guiding the attention computation in Transformer by using the structural bias in programs [25, 9, 38].
Source code summarization.
To generate a functional description for the given code snippet, code summarization models should capture both textual and structural information in programs.
Some early works [12, 30, 32] just treated the source code as a sequence or ﬂatten the AST using a structure-based traversal (SBT) method [10] and then performed a conventional seq2seq learning.
The hybrid works represented both textual and structural features separately, then a hybrid mechanism is used to combine the learned representations for the summary generation.
HDeepcom [11] applied the SBT method for structural representation and CAST [27] learning the structure via hierarchical splitting and reconstruction of abstract syntax trees.
Many Transformer-based methods [27, 1, 13, 3] used structure information to guide the attention computation.
TPTrans [25] encoded the pairwise path between code tokens and incorporated the path embeddings like relative position encoding.
Unlike the above-mentioned methods which view the source code as single-granularity data, we propose a multi-granularity source code summarization method.
Multi- scale feature learning.
The multi-scale feature processing is widely used in the computer vision community.
FPN [21] leverages the pyramid-shaped features of ConvNet and creates a feature pyramid that has rich semantics at all scales.
Recently, the multi-scale feature hierarchies were also introduced into Vision Transformer (ViT) by a variety of works like MViT [6], Swin [22] and
HRFormer [36], which show strong evidence that Transformer could work well with the multi-scale features.
Approach
As shown in Figure 2, given a source code, we ﬁrst construct sequences with three granularities and form them into a pyramid-shaped structure based on the mapping constraints between different granularities.
After that, the proposed pyramid attention mechanism (in Figure 3) is applied layer by layer in the encoding stage to produce a feature pyramid which is used by a vanilla Transformer decoder to generate the summary.
The details of the proposed pyramidal input constructor and pyramid attention mechanism are introduced in this section.
3.1 Pyramidal Input Constructor
The pyramidal input is constructed following the human code comprehension process described in
Section 1.
Entities in different granularities have their own emphases, telling speciﬁc attributes of source code from different viewpoints.
For instance, the coarse-grained statement sequence reveals the global logical semantics of code, the semi-grained token sequence shows the code grammar details, while the ﬁne-grained sub-token split from each token is attached with textual information.
We introduce the details of the proposed pyramid input constructor in the following.
Fine-grained sequence.
Each leaf node in the AST corresponds to a token (like boolean , value ) in the source code.
Since method names and variable names in programs are usually composed of multiple natural language words with CamelCase or snake_case , we can further split these combined tokens into sub-tokens, which reveal the textual information of source code.
To construct the ﬁnegrained layer (i.e., sub-token sequence), we ﬁrst obtain the leaf nodes sequence of AST by depth-ﬁrst pre-order traversal and then split each token into sub-tokens.
Such a sub-token technique is widely used by source code summarization methods [30, 25, 27] and proved to be effective.
Formally, let
[Figure]
Output
[Figure]
T =
[ t 1 , t 2 , . . . , t N ] ( T for text) denote the ordered sub-token sequence, that is, the ﬁne-grained input representation situated in the bottom of the pyramidal input.
For the example in Figure 1, T is [public, static, int, parse, String, As, Int, (, ...] .
Semi-grained sequence.
The attribute of the parent node of each token (i.e. leaf) in parsed AST represents its grammatical property.
We construct the semi-grained sequence by replacing the element in ordered token sequence with its grammatical property directly.
Formally, let G =
[ g 1 , g 2 , . . .
, g M ] ( G for grammar) denote the ordered token grammatical sequence, that is, the semi-grained input representation situated in the middle of the pyramidal input.
For the example in Figure 1, G is [modifier, modifier, type, identifier, (, ...] .
In addition, a one-tomany mapping relationship M G → T between token and sub-token is introduced.
M G → T ( g i , t j ) =[-
-]
if the sub-token t j is split from g i , which will be used to guide the information aggregation process from ﬁne-grained hierarchy to semi-grained hierarchy.
Coarse-grained sequence.
We use the nodes corresponding to statements to form our coarsegrained sequence, which reveals the logical information of the source code.
As the statement type is limited, we directly identify the AST node as a statement-level node if the tag belongs to the pre-deﬁned statement type set (see Appendix C for more details).
Formally, let L =
[ l 1 , l 2 , . . .
, l K ] ( L for logic) denote the sequence of the ordered statements, that is, the coarsegrained input representation situated in the top of the pyramidal input.
For the code in Figure 1, L is [declaration, parameters, try, return_statement, ...] .
Similar to the semi-grained sequence, we also construct a one-to-many mapping function M L → G between statement and token, which bridges the coarse-grained hierarchy with the semi-grained hierarchy.
M L → G ( l i ,g j )
= 1 if the statement l i is the most recent ancestor of token g j among L in AST.
In summary, we construct a sequence set {T , G , L} in three granularities which helps the model to capture patterns from different scales and extract mapping relationships {M L [Equation] G , M G [Equation]
T } between adjacent granularities which introduce inductive bias to aid the model to learn accurate semantics.
Thus, the sequences and the mapping relationships all together form the pyramidal input representation of the given code.
3.2 Pyramid Attention
The pyramidal layers {T , G , L} complement each other following the mapping relationships {M L → G , M G → T } .
We propose a novel pyramid attention mechanism for efﬁcient feature ag-
[Table]
[Table] [Table]
/42bNgdtfTDweG+GmXl+wqhUtv1tVFZW19Y3qpu1re2d3T1z/6An41QQ6JKYxaLvYwmMcugqqhj0EwE48hnc+5Obwr9/ACFpzDtqmoAX4RGnISVYaWloHrmPNABFWQCZG2E1JphlnTwfmnW7Yc9gLROnJHVUoj00v9wgJmkEXBGGpRw4dqK8DAtFCYO85qYSEkwmeAQDTTmOQHrZ7IHcOtVKYIWx0MWVNVN/T2Q4knIa+bqzuFEueoX4nzdIVXjlZZQnqQJO5ovClFkqtoo0rIAKIIpNNcFEUH2rRcZYYKJ0ZjUdgrP48jLpnTecZuPirllvXZdxVNExOkFnyEGXqIVuURt1EUE5ekav6M14Ml6Md+Nj3loxyplD9AfG5w/BxJcq</latexit>
G <latexit sha1_base64="w7GkJ7R/PBAv+q4CWvfdiGrNPwQ=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBIvgqsxIRZdFF7qsYB8wHUomzbShmWRIMkIZ+hluXCji1q9x59+YaWehrQcCh3PuJeeeMOFMG9f9dkpr6xubW+Xtys7u3v5B9fCoo2WqCG0TyaXqhVhTzgRtG2Y47SWK4jjktBtObnO/+0SVZlI8mmlCgxiPBIsYwcZKfj/GZkwwz+5mg2rNrbtzoFXiFaQGBVqD6ld/KEkaU2EIx1r7npuYIMPKMMLprNJPNU0wmeAR9S0VOKY6yOaRZ+jMKkMUSWWfMGiu/t7IcKz1NA7tZB5RL3u5+J/npya6DjImktRQQRYfRSlHRqL8fjRkihLDp5ZgopjNisgYK0yMbaliS/CWT14lnYu616hfPjRqzZuijjKcwCmcgwdX0IR7aEEbCEh4hld4c4zz4rw7H4vRklPsHMMfOJ8/fASRZQ==</latexit> input, and outputs a multi-scale representation without changing the original pyramid shape.
The whole process consists of a bottom-to-top aggregation pathway and a cross-granularity information interaction process.
gregation among these layers.
As illustrated in Figure 3, the pyramid attention module includes a bottom-to-top aggregation pathway and a cross-granularity information interaction process.
We describe the details in the following.
Bottom-to-top aggregation.
The bottom-to-top pathway does a ﬁne-to-coarse information aggregation, which ensures to enrich the higher-level representation using the lower-level semantics.
Given the pyramid input data: {T , G , L} and the mapping function {M L → G , M G → T } , we aggre- gate the representation of T into G w.r.t.
M G → T .
The formulation of the aggregation process is given as:
[Equation]
Where W agg ∈ R D × D , b agg ∈ R D are learnable parameters, g i , t
i ∈
R D are the i -th representation vectors in G and T respectively, D represents the dimension of the representation, and ! g i is the aggregated representation of g i .
The aggregation process from G into L w.r.t.
M L → G is the same as above.
We further conduct a aggregation directly from T into L w.r.t.
M L → T , which is obtained by:
[Equation]
In summary, we update {G , L} into { ! G , ! L} using T and {T , G} respectively.
The mapping-based aggregation is reasonable and necessary as the semantics of the element at high levels come from the combination of its descendants’ semantics.
The semi-grained sequence and coarse-grained sequence are initialized as the grammatical type of the tokens and statements.
Figuratively speaking, they are like torsos with empty skeletons, and we need to ﬁll them with textual information to make them complete.
There is an ablation experiment demonstrating its effectiveness in Section 4.3.
Cross-granularity interaction.
The aggregated semantics of the elements in semi/coarse-grained sequences tend to be local.
To alleviate this dilemma, we further make the elements in a speciﬁc granularity to query all elements at three granularities, which guarantees that the model can capture both ﬁne-grained and coarse-grained features.
Formally, we now have {T , ! G , ! L} after aggregation.
Taking sequence !
G ∈ R M × D of length M in certain granularity as an example, G will query all of {T , ! G [Equation] L} in the cross-granularity interaction process.
We ﬁrst concatenate the pyramid representation as
[Equation]
here N, M, K are the sequence lengths of T , ! G , !
L respectively. A relative position encoded multihead attention [26] is applied here and we use the relative position map R ! G ∈ Z M × |C| to make the model granular-aware.
The multi-head cross-attention (MHCA) process is calculated as:
[Equation]
where W hq , W hk , W hv ∈ R DH × DH , h ∈ [1 , . . .
, H ] are projection matrices of Query, Key and Value for h -th head.
W o ∈ R D
× D is projection matrix for attention output.
H represents the number of heads and ! X represents the output of multi-head self-attention process. W ! [Equation]
W !
[Equation]
∈ R DH [Equation] DH are projection matrices of Key and Value of relative position embeddings, the ! G means that they are speciﬁc to !
[Equation] [Table]
[Table] [Table]
L <latexit
[Table]
+J/npya6DjImktRQQRYfRSlHRqL8fjRkihLDp5ZgopjNisgYK0yMbaliS/CWT14lnYu616hfPjRqzZuijjKcwCmcgwdX0IR7aEEbCEh4hld4c4zz4rw7H4vRklPsHMMfOJ8/fASRZQ==</latexit>
For the construction of relative position map R , we calculate the clipped relative distance with max distance as 32 for the pair within granularity and assign the relative distance as a default value (e.g. max distance + 1 and max distance + 2) for all the pair across the speciﬁc granularity.
Figureshows a toy example of granular-aware relative position map with max distance as 3 for G crossgranularity interaction.
The cross-granularity interaction for T and L is similar to ! G .
We share attention parameters (i.e. W hq , W hk , W hv and W o ) across three granularities, but assign speciﬁc relative path parameters for each granularity.
In summary, in this cross-granularity interaction process, the semantics of the feature pyramid are updated as:
[Equation]
3.3 Pyramid Attention Transformer
As illustrated in Figure 2, the overall model is similar to the vanilla Transformer.
The self-attention module in the encoder is replaced with the proposed pyramid attention module for an efﬁcient process of the multi-scale input.
Notably, we only use the ﬁne-grained sequence in the pyramidal representation for decoding, which already contains enough semantics for summary generation and reduces the complexity of the model design.
Following prior works [30], we also apply a copy mechanism in our model which reduces the risk caused by out-of-vocabulary and speeds up training convergence.
Although there is overlap between the vocabularies of different granularities, the meanings of these common words are quite different when differentiated at different granularities.
For instance, else may be just a natural word in the textual sequence, but a keyword for control in a grammatical sequence.
Thus, we assign individual embedding layers for each granularity to avoid such confusion.
For efﬁcient parameter usage, a parameter sharing strategy is also applied in the cross-granularity interaction process as stated in Section 3.2, We ﬁnd the self-attention mechanism is strong enough to handle this multi-role mission.
Thus, our model doesn’t add too many parameters compared to
Transformer.
What’s more, such a parameter-sharing strategy also brings efﬁcient model implementation.
We only need to concatenate these three sequences and the relative position maps separately, then directly adopt a relative position encoded multi-head self-attention layer to implement the crossgranularity interaction module.
Experiments 4.1 Experiments Setup
Dataset and Metrics.
To demonstrate the effectiveness of the proposed method, we conduct experiments on two widely-used and well-developed java datasets: EMSE-DeepCom 2 [11] which is collected from GitHubs Java repositories and FunCom 3 [14] which has ∼ 2 million java methodcomment pairs.
We ﬁlter out the examples that cannot be parsed properly or are of too large size for both datasets.
And deduplication is done on EMSE-DeepCom to avoid data leakage.
Table 1 shows the statistics of the datasets.
[Table]
We use 6 metrics to evaluate the model performance on the datasets.
BLEU [24], Rouge-L [20] and Meteor [5] are widely used to evaluate the quality of text similarity in the domain of natural language generation.
Another 3 metrics are used to qualify the token-wise prediction performance.
We use beam search with a beam size of 3 for all models in the inference phase.
Data preprocessing.
All data preprocessing of code is based on the parsed syntax tree.
We obtain the AST using the open-sourced tool Tree-sitter 4 .
Please refer to 3.1 for details of the proposed pyramid data construction.
We re-implement the data preprocessing pipeline for all compared baselines based on the AST parsed by Tree-sitter, please refer to Appendix C for more details.
Baselines.
Early works viewed source code as a sequence of text and used the sequential model like LSTM or Transformer, we mainly compare with CODE-NN [12] and NeuralCodeSum [30] for this type of methods.
A few works only learned the structure features for this generation task, we compare with TreeLSTM [29] here.
Most works exploited the textual and structural information for this task.
One group of works learned to represent these two types of information separately, then a hybrid mechanism is used to combine them in the decoding phase.
We do comparisons with
HDeepCom [11], ASTAttnGRU [13] and CAST [27] for the hybrid fashion.
Another line of methods used structure relations as an inductive bias to guide the information interactions in the model.
Here, we compare with SiT [35], GREAT [9] and TPTrans [25].
For clarity, the backbone and taxonomy of each method are listed in Table 2.
And a copy mechanism is added for all transformer-based models.
We directly adopt the model implementation script if the code is publicly available, and re-implement the method according to the description in the corresponding paper if the code is not provided or not available.
What’s more, all baselines are integrated under a uniﬁed PyTorch-based code summarization framework developed by ourselves for fair comparisons.
We provide implement details for baselines in Appendix A.3.
Hyperparameters and training setup.
For fair comparisons, all the Transformer-based models use the default Transformer conﬁgurations with embedding dimension as 512, feedforward dimension as 2048, head number as 8, and layer number for encoder/decoder as 6 and all RNN-based
http:/ /https://
.github.io/tree-sitter/
[Table]
models use the hidden dimension with 512. For embeddings, we use the embedding dimension with 512 for all approaches.
The size of vocabularies is limited by 50000 over code sequences andover summary sequences in EMSE-DeepCom, and the corresponding conﬁgurations are 35000 and 30000 in FunCom.
All models are trained using NVIDIA Tesla A100 GPUs with a batch size of 64.
We train all baselines including our models using AdamW optimizer with a multi_step learning rate scheduler, and set the initial learning rate to 0.0002 and 0.003 for Transformer-based and RNNbased models, respectively.
And warmup strategy is used for stable training in the Transformerbased model.
The training epochs for each method are listed in Table 2.
4.2 Main Results
Comparisons with baselines.
The comparisons with baselines are listed in Table 2.
Without bells and whistles, we ﬁnd that our proposed PA-former substantially surpasses all baselines and achieves new state-of-the-art on both middle-size and large-size datasets.
Compared with NeuralCodeSum which is a relative positional encoded Transformer, our method has a 5% improvement in BLEU on the middle-size dataset.
For the current leading methods CAST and TPTrans, which both take all the nodes in ASTs and the sub-token sequence as input, our model only takes a subset of AST nodes and sub-tokens surpass both of them over 1.0+ BLEU scores on two datasets.
And thanks to the efﬁcient implementation and parameter sharing strategy, our model requires far less training time to converge, while TPTrans needs a very long time to converge as the large size pair-wise paths and the sparse scatter algorithm.
The leading results demonstrate that our multi-scale formulation for the task of source code summarization is reasonable.
To further evaluate the effectiveness of our method, we provide a case study in Appendix D. Moreover, we provide the results on RMSE-Deepcom dataset with error bars to show that the improvement achieved by our method is statistically signiﬁcant in
Table 11 at Appendix B.
Human evaluation.
As a complement, human evaluation is conducted over the transformer-based methods following the previous works [12, 27, 15].
We invite 5 volunteers with over 5 years of programming experience and ask them to score (0 - 10, 10 is the best) the generated summaries over 100 examples sampled randomly from the testing sets (20 from RMSE-Deepcom and 80 from Funcom).
Here we focus on two key points: naturalness (grammaticality and ﬂuency of the generated summary) and usefulness (what extent the generated summary is useful to understand the code).
We divide the 100 examples into 5 groups (each one has 20 examples) and assign each group into [Table] different volunteers, and the ﬁnal score of each example is the average of its 3 "review" scores.
As shown in Table 3, our PA-former outperforms others in both naturalness and usefulness.
4.3 Ablation Study
In this part, we analyze the importance of designs in our proposed PA-former using a series of ablation studies on EMSE-DeepCom dataset.
Unless otherwise speciﬁed, the training setup is the same as above.
Multi-scale v.s. single-scale.
To demonstrate the effectiveness of multi-granularity formulation, we degenerate the pyramid input into a single large sequence by concatenating the three granularity sequences and directly feed the same sequence into a relative positional encoded Transformer with a copy mechanism.
As shown in Tabel 4, more data in the input sequence obtains performance improvement compared to NeuralCodeSum which only takes a sub-token sequence as input.
However, processing the same data using our proposed multi-scale way achieves higher performance.
[Table]
Effectiveness of pyramid attention.
The core components of pyramid attention are the bottomto-top aggregation and cross-granularity interaction.
We show the necessity for both of these two components here.
To demonstrate the effectiveness of bottom-to-top aggregation, we evaluate a variant of PA-former whose bottom-to-top pathway is disabled.
As shown in Table 5, the model performance drops by about 1.1 BLEU scores without such a component.
To demonstrate the effectiveness of cross-granularity interaction, we evaluate a variant of PA-former whose cross-granularity interaction is disabled which directly degenerates into NeuralCodeSum as we only use the ﬁne-grained encoder output for decoding.
As shown in Table 5, the model performance drops by about 1.7 BLEU scores without such a component.
Such results reveal the importance of cross-granularities interaction.
We further disable the granular-aware relative embedding to investigate its importance, and we observe a big drop in performance shown in Table 5, which shows its effectiveness.
[Table]
Decoding strategy.
The output of our PA-former encoder is a pyramid representation, thus there are various ways for decoding.
Our PA-former directly uses the ﬁne-grained sequence just for simplicity.
Here, we study other implementations that utilize all three granularity of representation.
The most intuitive method is concating the three granularity representations, and another way is using them in a serial strategy [18].
As Table 6 shows, using the ﬁne-grained representation is good enough in this task to beneﬁt from the design of information interaction across granularities.
The results are a bit counterintuitive, we think a reasonable explanation is that the ﬁne-grained representation already contains enough information beneﬁted from the proposed pyramid attention.
When all three granularities are used, there is severe information redundancy between different granularities, which complicates the task.
[Table]
Grammar input.
When constructing the pyramid input, we intentionally enforce the bias between different granularities by assigning different types of content.
For example, we bias the semi-grained sequence into grammar features by assigning token grammar types to it.
To evaluate the rationality of assigning different types of content, we compare the proposed pyramid input without grammar information, which set the initial content of semi-grained and coarse-grained sequences to empty.
The results in Table 7 prove the rationality of pyramid input design.
[Table]
Conclusion
In this paper, we present a multi-scale formulation for source code summarization.
Given a code snippet, we construct a pyramidal input with three granularities.
And a novel pyramid attention mechanism is proposed for efﬁcient information interaction across granularities.
Without bells and whistles, the proposed method obtains 5% improvement over the strong baseline and achieves new state-of-the-art results.
Additional ablation studies show the effectiveness of the proposed modules.
Furthermore, our multi-scale formulation for the source code can be easily extended to other codeunderstanding tasks.
In future work, we would try to scale our model up and pre-train it on larger source code corpora to further improve its capability and generalizability on source code comprehension tasks.
Acknowledgement
This research was supported by NSFC (62076121, 61921006).
The authors would like to thank Ying
Li, Hao-Yuan He, Ren-Biao Liu, Yu-Xi Sun and Zhi-Yu Shen for helpful discussions.
Finally, We are grateful to all reviewers for their insightful comments.