C ODE S UMMARIZATION : D O T RANSFORMERS R EALLY U NDERSTAND C ODE ?
A BSTRACT
Recent approaches for automatic code summarization rely on fine-tuned transformer-based language Models often injected with program analysis information.
We perform empirical studies to analyze the extent to which these models understand the code they attempt to summarize.
We observe that these models rely heavily on the textual cues present in comments/function names/variable names and that masking this information negatively impacts the generated summaries.
Further, subtle code transformations which drastically alter program logic have no corresponding impact on the generated summaries.
Overall, the quality of the generated summaries even from state-of-the-art (SOTA) models is quite poor, raising questions about the utility of current approaches and datasets.
I NTRODUCTION
Code summaries play an important role in program understanding, maintenance and debugging.
Recent work (Ahmad et al., 2021a; Wu et al., 2021; Z¨ugner et al., 2021; Phan et al., 2021;
Qi et al., 2021; Elnaggar et al., 2021) towards automated code summarization adopts two primary approaches: (i) Fine tuned Language Models (LM) or (ii)
Deep models that inject Program Analysis Information (PAI) claiming to facilitate better understanding of program semantics.
In this paper, we perform an empirical analysis to evaluate the code understanding capabilities of these models for summary generation.
We assume that to generate code summaries the models need to understand the underlying semantics of the code.
Based on this assumption, we hypothesize that semantics preserving code transformations would have less effect on code summaries, whereas code transformations that change the underlying logic of the input code would alter the summaries.
Conversely, our observations lead to rejection of these hypotheses.
Following observations of our studies may prove useful for the code summarization research community:
The BLEU scores of existing code summarization models reported on public datasets are very low (in the range of 11.17 to 26.53) (Wang et al., 2021), especially for out-of-domain data (in the range of 5.45 to 7.85) where train and test codes belong to distinct projects (Liu et al., 2020).
This calls into question the utility of these models for real-life applications.
Testing the models on codes with semantic preserving transformations by replacing meaningful function and variable names with generic vocabulary words, negatively impacts the BLEU score (average drop of 7).
This is not only true for the LM based models but also for the models which claim to understand the program structure by injecting PAI.
This likely points to prevailing summarization models performing ‘short-cut’ learning by primarily relying on the inductive biases from meaningful function and variable names.
Such heavy reliance may be acceptable for domain specific application codes, where the developer ensures the usage of meaningful function and variables names.
However, when code summarization heavily depends upon the underlying algorithm or logic implemented in the code such reliance would be detrimental for code comprehension.
Training with codes after semantic preserving transformations leads to no improvements in BLEU over the original models.
This indicates that the models are extremely reliant on textual cues and are unable to understand the underlying logic of the code when these are removed.
This highlights the
[Table]
if (entity is not None): memcache.set(entity key, entity) return entity def func(var1):var2=memcache.get(var1) if(var2isnotNone):returnvar var
=ndb.
Key(urlsafe=
var1)var2=var3.get
() if(var2isnotNone):memcache.set(var1,var2) returnvar
Summ.
Model
SIT
PLBART GT get entity from memcache if available.
get entity from memcache if available.
Original returns a key that can be used for entity.
retrieves an entity from memcache.
EXP-Te-SPT returns an instance of c. return the value of var2.
EXP-Tr-SPT cache keys for azure entities.
retrieves an entity from memcache.
EXP-TrTe-SPT cache keys in memcache.
returns the value of a memcache key.
EXP-Te-SDT returns a key that can be used for entity.
retrieves an entity from memcache.
need for designing better training strategies to facilitate the understanding of the code logic, such as self-supervision with both semantic preserving and disrupting transformations. 4.Transformations which change the semantics of the code in terms of perturbing the operators in expressions (arithmetic and logic) and thus impacting the high-level logic of the code, have very minimal impact on the BLEU scores (average drop of 0.13), demonstrating that the models are not paying much attention to code semantics while generating the summaries.
Getting rid of the leakages in the datasets in terms of having exactly same code-summary pairs in the splits leads to a large drop in the BLEU scores (average 11), highlighting the need for carefully designing datasets. Datasets should completely avoid code overlaps across the splits not only in terms of codes having same surface forms but also codes which belong to a same project sharing semantics.
Datasets should also facilitate learning of code semantics and prevent over reliance on textual correlations.
R ELATED W ORK 2. C ODE S UMMARIZATION D ATASETS
Publicly available datasets such as TL-CodeSum (Hu et al., 2018b), Python (Wan et al., 2018),
Funcom (LeClair et al., 2019), CCSD (Liu et al., 2020), CodeSearchNet (Husain et al., 2019) and
CodeXGLUE (Lu et al., 2021) have function-summary pairs collected from open source GitHubrepositories.
These datasets have the following lacuna:
1.Having code comments as a part of the source code: CodeSearchNet (Husain et al., 2019) have code comments in the codes and need pre-processing to avoid biases.
For example, the code1 https://github.com/
[Table]
summary pair of Java in CodeSearchNet depicted in example (a) of Table 3, has comments in the code which have textual correlations with the summary.
Data leakage: TL-CodeSum (Hu et al., 2018b) and Python (Wan et al., 2018)
datasets have data leakages with duplicate code-summary pairs across train and test splits.
Examples (b) and (c) in
Table 3 depict example Java and Python code-summary pairs present in the train and test splits of the mentioned datasets.
Meaningful function and variable names having textual correlations with the words in the summary: As depicted in Table 1, the code snippets in the current datasets have meaningful function and variable names that have textual correlations with the Ground Truth (GT) summaries, leading to an inductive bias.
Highly abstract summaries that are divorced from the code logic: As collected from Github repositories, the summaries of existing datasets (Table 1) are in the form of code-comment pairs where the code snippets are at function-level.
For models to learn the underlying program logic, we need the code-summary pairs in the form of complete code with more abstract code-level summaries.
For example the code-summary pair from CodeNet(Puri et al., 2021)
in example (d) in Table 3, provides a problem description of the complete code summarizing the underlying logic of the code.
Domain specific summaries that are not obtainable from code requiring external knowledge beyond the
code logic for summary generation: In the CodeNet dataset the problem descriptions come from a variety of domains.
Thus, it is impossible to predict the domain-specific components of the summaries from the codes as an input, which require external domain knowledge.
For example, from the code illustrated in example (d) of Table 3, to generate the illustrated GT summary external domain knowledge in terms of the meaning of ‘parallel lines’ (lines having same slope and the definition of slope computation) is required.
No out-of-domain splits: As the existing datasets may not have domain overlaps, models trained on one dataset do not perform well on the other (out-of-domain data) as depicted by the codes in examples (e) Python and (f) Java in Table 3 from CodeNet and the corresponding GT and predicted summaries by PLBART trained on CodeSearchNet.
Since there no domain overlap between these datasets, the predicted summaries do not match with the GT summary and most of the time are meaningless.
No datasets for legacy programming languages like COBOL:
The above listed code-summary dataset addresses only high-resource programming languages such as Python, Java, Javascript, PHP, Ruby, Go and C#.
For practical applications, where there is a need to maintain and debug legacy codes we need datasets that would facilitate summarization of legacy languages such as COBOL.
We perform our analysis on CodeSearchNet, TL-CodeSum and Python datasets for Python and Java programming languages.
The data statistics are provided in
Table
2. C ODE S UMMARIZATION A PPROACHES
Neural code summarization approaches utilize one of the following : (i) Language Models (LM) pre-trained with monolingual programming data and further fine-tuned with code summary pairs or (ii) Deep models (Transformers, LSTMs, Graph Neural Networks) exploiting program analysis information in terms of Abstract Syntax Trees (ASTs), data dependencies and/or control flows to incorporate code semantics.
LM based approaches such as PLBART (Ahmad et al., 2021b), CodeT5 (Wang et al., 2021), CoText (Phan et al., 2021), ProphetNet-Code (Qi et al., 2021), CodeTrans (Elnaggar et al., 2021), and CodeBERT (Feng et al., 2020), pre-train a LM on mono-lingual programming language data collected
[Table]
,a2,a3,...,an,findthemaximumsumofacontiguoussubsequenceofthosenumbers.
Notethat, asubsequenceofoneelementisalsoacontiquoussubsequence.
Theinputendwithalineconsistingofasingle0.
Writeaprogramwhichreplaceallthelower-caselettersofagiventextwiththecorrespondingcaptitalletters.
Printtheconvertedtext.
PLBART Summary Readsinputandprintsthemaximumvalueoft.
Themainentrypointforthisclass.
Approaches exploiting Program Analysis Information (PAI) use LSTMs (Hu et al., 2018a; Alon et al., 2018; LeClair et al., 2019),
Transformers (Ahmad et al., 2020; Wu et al., 2021; Z¨ugner et al., 2021; LeClair et al., 2019; Zhang et al., 2020), Graph Neural Networks (GNNs) (Liu et al., 2020; LeClair et al., 2020; Wang et al., 2020) or a combination of these (Choi et al., 2021; Shi et al., 2021) and inject PAI in the form of Abstract Syntax Trees (ASTs), data dependencies and/or control flows. The PAI is provided in the form of flattened ASTs using pre-ordered or structure based traversal (Hu et al., 2018a; Alon et al., 2018; LeClair et al., 2019), pre-defined adjacency matrices with the edges as an inductive bias for the attention between nodes (tokens) (Wu et al., 2021), relative positional encodings between adjacent nodes (Ahmad et al., 2020; Z¨ugner et al., 2021) or feeding the Code
Property Graphs (CPGs) to the model (Liu et al., 2020).
Some studies also enhance these models by incorporating information retrieval techniques (Li et al.
; Zhang et al., 2020; Liu et al., 2020), where the prototype summaries of similar codes are retrieved from a database and are edited by using an encoder-decoder setting.
For our analysis, we include one model from each of the above categories, namely PLBART (Ahmad et al., 2021b) and Structure Induced Transformers (SIT) (Wu et al., 2021).
2. A DVERSARIAL P ROGRAM P ERTURBATIONS
This section throws some light on the literature which explores the idea of investigating adversarial program perturbations to break the AI models built for code intelligence tasks.
For program synthesis task, with NL as an intent Liguori et al.
(2021), shows that parsing NL with generic variable names achieves better synthesis.
Karmakar & Robbes (2021) probe the codes to evaluate if the code representations provided by the pre-trained models encode the surface level, syntactic, structural and semantic code characteristics and points at the need for designing better pre-training strategies.
Yefet et al.
; Rabin et al. (2021) defines an approach which learns adversarial examples having semantic preserving transformations, specifically variable renaming and adding dead code to the original program to attack the trained neural model to make incorrect predictions for the tasks of bug finding and predicting method names.
Bielik & Vechev (2020) checks the robustness of neural models trained for predicting program properties by creating adversarial examples.
As opposed to these approaches, we experiment with semantic preserving transformations for the code summarization task to evaluate the model dependency on meaningful function and variable names for summary generation.
Srikant et al.
(2021) identify the perturbations in terms of replacing the existing code tokens and the sites in the program to apply the perturbations to break sequence-to-sequence code summarization models.
However, the summaries taken into consideration are of only few ( ∼ 1-3) words.
On the other hand, we aim at exploring the effect of semantic preserving along with semantic disrupting transformations on the predicted summaries, which are more elaborate in terms of token lengths.
Based on the assumption that the summaries require the models to understand the underlying logic of the code and can not just rely on exploiting the inductive bias created by the meaningful function and variable names, we hypothesize that the semantic preserving transformations should have no major effect on the generated summaries, whereas the semantic disrupting transformations should negatively affect the summaries.
T RANSFORMATIONS
We perform causal analysis by tweaking the code using the following transformations to preserve or change code semantics and then observe the effect on the resulting summary and BLEU scores.
Table 1 demonstrates the transformations.
SPT are the set of S emantic P reserving T ransformations, which include (i) CC removing the Code
Comments from 17% of the codes in CodeSearchNet (ii) FN replacing meaningful user-defined
Function Names with more generic (but unique) function names, and (iii) V N replacing meaningful user defined local Variable Names with more generic variable names, unique per existing variable name, such that data-dependencies are preserved.
Generic names carry no semantics and are selected from the existing model vocabulary.
FN and V N are applicable to all codes in all the datasets.
SDT s are the set of S emantic D isrupting T ransformations, which include (i) replacing an arithmetic and relational operator with its inverse (For example, replacing + with − or equality =
= with inequality !
= , etc) and (ii) replacing a logical operator with its complement (For example, replacing AND with OR ) such that the code execution is not hampered but the semantics of the code is disrupted.
Changing the operators would bring in change in the control flow dependencies [Table] of the code and thus would lead to structural code changes.
∼ 78%, 68%, 40% and 43% of codes in
CodeSearchNet-Java and Python, TLCodeSum and Python datasets, respectively, are modified with
SDT .
The intent is to observe the change in BLEU, by comparing the summaries generated by the models with the transformed and original codes, against the original GT summaries, which are retained for both the transformations.
E XPERIMENTAL S ETUP
This section explains the performed experiments and the corresponding hypotheses in detail.
EXP-Te-DL
We address the Data Leakage (DL) in the datasets by removing 38.49% Java and 21.66% Python code snippets from the Test Set of TL-CodeSum and Python datasets, that exactly matches with the code snippets in the train set resulting in inflated BLEU scores.
We expect a drop in average BLEU scores after filtering these samples from the test set.
We use this filtered test set for the following experiments
EXP-Te-SPT Models trained on the original train data are tested on the SPT transformed Test Set.
We hypothesize that the model BLEU scores should not change from unmodified trainset-testset scores
as SPT s are
semantic preserving.
EXP-Tr-SPT Models trained with the SPT transformed Train Set are tested on the original test data.
Since the model can no longer exploit function and variable names to generate summaries, this experiment should test whether
the model
is capable
of understanding the program logic and
if
so, we hypothesize that the BLEU
scores should improve. EXP-TrTe-SPT Models trained with the SPT transformed Train Set are tested on the SPT transformed Test Set.
Along similar lines of EXP-Tr-SPT we expect improvements in the BLEU scores
over unmodified trainset -testset results indicating that the model better understands code. EXP-Te-SDT Models trained on the original train data are tested on the SDT transformed Test Set.
As the SDT changes the semantics of the programs, based on the assumption that the model uses code semantics for summarization, we hypothesize that the generated summaries are different from the original GT summaries, leading to a drop in the BLEU scores. EXP-Tr-SDT Models trained on the SDT transformed Train Set retaining the original GT sum- maries are tested
on the original Test Set.
As SDT changes the code semantics, mapping such programs to the original summaries lead to noisy training. Thus, we hypothesize that the generated summaries on the original Test Set would be noisy leading to some drop in the BLEU scores. EXP-TrTe-SDT Models trained with the SDT transformed Train Set and Tested with SDT transformed Test Set, both retaining the original GT summaries Both Train and Test sets being noisy
, this
set-up
does not
provide
any
insights
about
change
in the model performance after probing.
Hence we do not perform this experiment.
To programmatically transform the codes, we use javalang 3 and ast 4 packages.
We detect the function and variable names by AST construction.
The logical and arithmetic operators are detected by using regex 5 .
SIT 6 is trained on TL-CodeSum and Python dataset and PLBART on CodeSearchNet.
For having comparisons across the models, we fine-tune pre-trained PLBART 7 with TL-CodeSum and Python, where the codes are tokenized using the Tree-sitter tokenizer 8 .
For fair comparison, we retain the hyper-parameter settings (Ahmad et al., 2021a; Wu et al., 2021) and run the experiments on one Nvidia Tesla V100 32 GB GPU.
SIT and PLBART take ∼ 34 and 8 hours to train.
Experiments on CodeSearchNet are performed with only PLBART as the program analysis information required for the SIT model is not available for this dataset.
R ESULT AND A NALYSIS
Table 1 illustrates the examples of Java and Python codes from TL-CodeSum and Python datasets and the corresponding transformed code with SPT and SDT .
However, it should be noted that, we never perform both transformations simultaneously.
EXP-Te-SPT summaries do not match with the GT and are inferior to the original model summaries, showcasing the negative influence of SPT .
EXP-Tr-SPT and EXP-TrTe-SPT summaries are closer to the GT as compared to EXP-Te-SPT demonstrating the positive effect of an SPT transformed train set.
Subtle change in the operators of ‘for loop’ in the Java code with EXP-Te-SDT as demonstrated in Table 1, completely disrupts the code logic.
However, no change in the corresponding summaries showcase no influence of SDT .
Table 4 illustrates the smoothed BLEU-4 scores for all the experiments.
As expected, EXP-Te-DL showcases substantial drop in BLEU (average 11) after removing data leakage.
Comparable BLEU scores for SIT and PLBART models questions the benefit of infusing PAI into the model as opposed to using a fine tuned LM.
As CodeSearchNet has no data leakage, there are no drops in the BLEU with EXP-Te-DL .
With EXP-Te-DL , the overall BLEU scores are in the range of 16-24, questioning their utility for real-life applications 9 .
There is a further drop in BLEU (7.09) with EXP-Te-SPT showcasing the huge role comments and meaningful function/variable names are playing in summary generation.
The ablation experiments demonstrate that function names have the most impact on generation followed by variable names and comments leading to 4.99, 2.85 and 0.35 average drops in BLEU score.
The drops in the BLEU scores with EXP-Tr-SPT (2.62) and EXP-TrTe-SPT (3.27) are less as compared to that of with the
EXP-Te-SPT proving that training with more generic function and variable names is helping the model to better understand the semantics.
However, no improvements in BLEU over EXP-Te-DL demonstrates the need for designing better pre-processing and training strategies for the task.
With
EXP-Te-SDT the drops in BLEU are very minor (0.13) showcasing that the transformations which change the semantics of the code ( SDT ) have no effect on the summaries and thus it is questionable if the models are paying any attention to the logic/semantics of the code.
With EXP-Tr-SDT the model trained with noisy pairs of data ( SDT transformed code with original summary) showcase random behaviour with drop in BLEU for most of the cases, whereas for others we can observe a very slight increase in BLEU.
We perform qualitative analysis for the sampled test cases fed to the SIT model trained on Python dataset Table 5.
High drop in BLEU for EXP-Te-SPT and EXP-TrTe-SPT is due to (i) GT summary having textual correlations with the function and variable names in the code.
Training with original data leads to reliance on such correlations which are not available to exploit due to SPT transformed test data (Example A) (ii) Model trained with SPT transformed training data removes the bias of textual correlations.
Such model, during inference, tends to copy the summary of a syntactically similar transformed code from the training set (Example B). No drop in BLEU for EXP-Te-SDT is due to GT summary having high reliance on textual cues in the code leading to model not learning to pay attention to the operators present in the code (Example C). [Table] , though widely used as a metric for code summarization (Shia et al., 2022), is not an appropriate metric to to measure the quality of the code summaries (Allamanis et al., 2018).
BLEU only assigns credit to exact n-gram overlaps and does not take the sentence structure and the semantics into account.
More importantly, BLEU is uncorrelated with the programmers comprehension of the source code (Stapleton et al., 2020) and the program correctness (Austin et al., 2021; Chen et al., 2021;
Hendrycks et al., 2021).
BERTscore (Zhang et al., 2019) or BARTscore (Yuan et al., 2021) metrics take sentence fluency and semantics into consideration.
However, these metrics have not been evaluated in the context of source code summarization.
The above analysis can be better supported by human evaluation (Shia et al., 2022; Zhu & Pan, 2019).
However, considering the size of the test data (Table 2) and the experiments performed (Table 4), human evaluation is infeasible.
Thus, the choice of BLEU score as the evaluation metric is the limitation of this paper.
This points to the need of an appropriate evaluation metric for the task.
We also plan to perform human evaluation on sampled data as the future work.
Through empirical studies of SOTA code summarization models, we demonstrate the negative impact of semantics preserving code transformations on the generated summaries.
Additionally, we demonstrate that semantic disrupting transformations leave the generated summaries largely unchanged.
This questions the code understanding capabilities of these models and points to the need for better training strategies and well-curated datasets to facilitate code understanding.
The SPT and SDT transformations devised here offer some ideas for potential self supervised strategies to better train these models.
The current analysis is restricted to a subset of code-summary datasets, programming languages, neural models and the transformations.
We are working on extending it to generalize our observations.