Paper: A Critical Study of What Pre-trained Code Models (do not) Learn (ICLR-2024)

Reviewer comment (Evaluation):
To my understanding, most of the studied models are primarily token-based models. It is somewhat unexpected by construction that the models will have accurate knowledge of AST and DFG. Also, depending on the tasks it may not matter very accurate knowledge of these code properties. For example, many previous studies have shown that for code summarization tasks, a good choice of variable names is good enough. It is not very clear from the paper what kind of properties primary token-based models are expected to learn and how they have learned it.

Author response:
These models indeed act on tokens but they do not just take in some tokens and split out a new one. They are optimized to perform some task. There is an implicit assumption that structure in input data can be learned by the models from simple inputs with proper optimizations. Evidence has been provided to support this assumption by multiple studies in different fields, including NLP and vision. Previous work in NLP has shown that with tokens as inputs and proper optimization, models can learn the language grammar. Similarly, code models have been shown to learn program syntax. Our work is an attempt to understand how much structure can be encoded in models with optimizations based on tokens. So, we believe it is not correct to say it should be unexpected using construction. 

    Our work points out a significant limitation in learning these structures about code, which has been previously unexplored. We do not claim that the learning should be perfect; we only point out that some very important relations necessary for comprehending code syntax and code logic are not encoded by the models in the self-attention values and hidden representations.

    Moreover, not all models are trained only with tokens. GraphCodeBERT takes data flow graph nodes as inputs along with tokens, while, UniXcoder takes nodes of flattened AST as input.
 
    The reviewer rightly points out that for code summarization a good choice of variable names is good enough. However, we argue that such dependence on function names as opposed to code logic for summarization is a symptom of shortcut learning and hinders generalization. We have added a discussion on this argument in Section 4.3 of the updated paper as well as in the common comments to all reviewers. We kindly ask the reviewer to consider these too.

Reviewer comment (Theory):
Why do authors expect the self-attention score between syntactic and identifier nodes to be high?

Author response:
We assume self-attention between syntactic and identifier tokens to be high only if they are within the same motif structure. This assumption is based on results in previous works such as Wan et al (2022). Further, the relation between syntactic tokens and identifiers plays an important role in program flow. For example, `if` and `else` defines code blocks but the flow of the program is decided by the identifiers related to these syntactic tokens.

Reviewer comment (Presentation):
Overall, the paper's presentation is poor. It does not have a clear and logical structure, which poses difficulty in reading and understanding. The visualization is not clear. Explanations of the figures and data are not detailed enough, so it is a bit hard to relate them to the conclusions.

Author response:
We have made changes to the paper described in general comments to make the flow and the conclusions of the paper more clear. We have also modified the captions of Figures and Tables to add additional details.

Reviewer comment (Writing):
While the use of the “motif structure” seems integral to the construction of the code graph, it is not explicitly defined in the work; providing some background or context for this concept may be critical.

Author response:
The term 'motif structure' was defined in Wan et al (2022) and we use the definition as is. The citation was missing in the Introduction section. Following the suggestion of the reviewer, we have described the motif structure in Appendix B of the updated paper and have cited the relevant work in the Introduction.

Reviewer comment (Theory):
The paper makes some unsubstantial claims. For example, in the intro, they say one group of papers says LLMs are working for code, while some other group of papers find cases where LLMs fail. The authors say these two trends are contradictory. I think this is how any field progresses ---they are not necessarily contradictory trends.

Author response:
Based on the suggestions of the reviewer, we have made changes to the Introduction section to make our claims clearer and have also reworded the text about contradictory trends.

Reviewer comment (Theory):
Why do you set all the values above threshold to 1? Why not maintain an individual attention score?

Author response:
We had initially set the values to 1 to replicate previous work and then continued our analysis with this value to ensure our work remained comparable to those works. We can use the individual attention values by weighing all computations with the original value. Since the attention values are between 0 and 1, it will make the limitations of the models more stark. But, it will not change the conclusions drawn from the analysis. Setting it to 1 shows that the limitations remain even if the attention is very high. We have added this explanation in Section 3.2.1 of the updated paper.

Reviewer comment (Experiments):
Currently, the analysis was performed on 5 PCMs that were proposed in 2020-2022 and are relatively small. No investigation of more recent models, especially LLMs (such as CodeLLaMa and Starcoder), is performed. It is not clear if the obtained findings are applicable to more recent, large models.

Author response:
Selection of models: We agree that the paper focuses on slightly older and smaller models. However, we believe that the analysis of these models is still relevant and provides important insights for future research. We provide our argument for selecting these models in common response to all reviewers and also in Appendix C of the updated paper.

Reviewer comment (Novelty):
It seems that this work closely aligns with the research conducted by Wan et al. (2022) and places a greater emphasis on the fine-grained details of code tokens, achieved by categorizing input tokens into syntactic tokens and identifiers. From this standpoint, this work is considered a bit incremental.

Author response:
Incremental work: We disagree with this comment. While the initial motivation was to replicate the work by Wan et al. (2022), we found that the results and conclusions of that work were significantly affected by the choice of assumptions that were made. Subsequently, we did a comprehensive analysis of existing works in this domain and found that such assumptions abound.  We, then, extended previous works to point out the limitations of current models. We used DirectProbe, which has not been used for PCMs before. For analysis with DirectProbe, we did a complete experimental design and created a dataset to run DirectProbe. We also extended the existing analysis to more models and data flow graphs. In previous work, data flow graphs were only used with probing-based classifiers with an assumption of linearity in encoding. Our work revealed that the linearity assumption is not valid. Similarly, our work is the first to use graph edit distance for attention analysis, which provided new insights into the limitations of PCMs.

Reviewer comment (Theory):
The authors claimed that their findings shed light on why PCMs fail to generalize beyond dataset they are trained on and in real world applications. I do not think this statement can be well supported by the findings shown in this paper. The authors investigate the inability of pre-trained code models to comprehending relations across the syntactical keywords and identifier tokens, however, the connections between the comprehending ability and the generalization ability are non-trivial and need to be explained explicitly.

Author response:
We agree with the reviewer that the original paper did not discuss how our findings shed light on why PCMs fail to generalize beyond dataset. We have now added details explaining the relation between the ability of models to comprehend code logic and syntax and their generalization ability in Section 4.3 of the update paper.

Reviewer comment (Presentation):
Several statements in this paper are not consistent. For example, the research objectives are inconsistent, the study attempts to explore the extrapolation ability of a model in the introduction, however, the methods mainly focus on evaluating the ability of a model distinguishing syntactical tokens and identifiers.

Author response:
The research objective of the paper is to understand the limitations of PCMs with respect to what code information they do not encode. We have stated this in the Abstract as well as in the Introduction. We had additionally claimed that the limitations we found shed light on the limitations of models to generalize. However, we had not made this connection clear in the paper. We now do so in Section 4.3 of the updated paper.

Reviewer comment (Experiments):
Are the obtained findings applicable to more recent, large models?

Author response:
This cannot be answered without evaluating the larger models. However, a comparison between PLBART and CodeT5, two encoder-decoder models with different sizes, did not reveal any difference in the model's ability to understand syntax better only because of size. Moreover, as we explain in Appendix C of the updated paper, analysis of smaller models also reveals relevant insights.

Reviewer comment (Experiments):
The scope of the models and datasets analyzed is limited, and the paper could be strengthened by considering a wider range of model architectures, sizes, and training data sources to provide a more comprehensive understanding of the limitations across various PCMs.

Author response:
1. Limited choice of model architectures, sizes, and data sources: 
 
     We have explained the limitations regarding model architecture in Appendix C of the updated paper and also mentioned it in common response to all reviewers.

    The limited choice of the dataset was deliberate to remove any effect of data distribution shift. All models except PLBART were trained on the CodeSearchNet (CSN) dataset. Also, CSN is a fairly large dataset with 6 programming languages and 2 million comment-code pairs. PLBART was trained on Java and Python code from Google BigQuery. Despite a different dataset and the potential for distribution shift, we considered PLBART due to its smaller size compared to other encoder-decoder models. 

    PLBART is about half the size of encoder-decoder models such as CodeT5, but the difference in sizes does not seem to affect the conclusion. Moreover, the idea behind analyzing smaller models was to understand the limitations of smaller models, to improve them instead of always going for larger sizes. We have added a discussion on the use of smaller models in Appendix C of the updated paper.

Reviewer comment (Experiments):
How do the observed limitations in PCMs affect their performance on real-world tasks? It would be helpful to provide examples or case studies to illustrate the practical implications of these limitations.

Author response:
1. Thank you for this suggestion. We have updated the section on the limitations of PCMs (Section 4.3 in the updated paper) to discuss how the limitations affect PCMs on real-world tasks and in generalization beyond training data. We have also added these explanations in common response to all reviewers.

Reviewer comment (Experiments):
Are the identified limitations specific to the Transformer-based models studied in the paper, or do they extend to other types of models, such as RNNs or LSTMs, in the context of code understanding?

Author response:
2. We cannot comment on whether these limitations extend to RNNs / LSTMs as well without analyzing code models with RNN / LSTM architecture. Our focus in this paper was on models based on transformer architecture. We agree with the reviewer that extending this work to RNNs / LSTMs would be very interesting.  
Unfortunately, there are very few works on code models which use RNNs/LSTMs with attention and even fewer with self-attention. More importantly, though, the way self-attention works in RNNs is different from that of transformers. The attention in RNN is over hidden representation at different time steps. Hidden state H_t has seen all tokens till t, and so is not independent for each token. Moreover, the self-attention matrix is not always the same size as the input. For example, [3] has an attention matrix of size r * n for an input of n tokens. Our work assumes an n * n attention matrix, which is valid for transformer architecture, and so cannot be directly extended to RNNs / LSTMs. Finally, the existing RNN / LSTM models are trained on much smaller datasets.
This said, inspired by this helpful suggestion, in future, we will look at modifying self-attention-based LSTM architecture to have n * n attention matrix or train recurrence-based transformer models such as [4] on code and analyze and compare them with transformers. This would require modifying some models, training them with code datasets such as CSN and with different inputs and objectives. We feel this would be a research in itself and should be considered a topic of another paper.

Reviewer comment (Experiments):
Have the authors considered exploring the impact of different pre-training objectives or methods that could potentially address the limitations identified in the paper? Some suggestions or proposals to improve the training process would be valuable.

Author response:
3. We did consider the impact of different pre-training objectives and had made suggestions in Section 4.3 of the original paper. We discuss these suggestions in Section 4.4 of the updated paper and have made changes to the text so that the impact of pre-training objectives and the suggestions are hopefully stated more clearly.

Reviewer comment (Experiments):
How do the limitations in the PCMs affect their ability to learn from more diverse training data, such as codes from different domains or programming languages? Would incorporating such variety during the training phase help in mitigating the observed limitations?

Author response:
4. This is a very important point, which is already considered in the paper. All the models studied in the paper have been trained on multiple languages. PLBART has been trained on Python and Java, while the others have been trained on the 6 languages present in the CSN dataset. Moreover, CSN is a huge dataset with 2 million comment-code pairs. Thus, the limitations exist despite the variety in training data. We have modified Appendix E of the paper to include these details.

Reviewer comment (Experiments):
The selection of models analyzed is limited. While the paper has covered 5 models that are either encoder-only or encoder-decoder, it has not covered decoder-only PCMs, which seem to be the major choice of pre-trained language models currently.

Author response:
Since this weakness was mentioned by multiple reviewers, we have explained how the models we have analyzed provide important and relevant insights in the common response to all reviewers and also in Appendix C.

Reviewer comment (Experiments):
Lack of real-world evaluation. The paper focuses on analyzing model internals. It does not evaluate how the limitations discovered actually affect model performance on downstream tasks. Testing on real-world code generation and code search benchmarks could better highlight the practical implications.

Author response:
Since this weakness was mentioned by multiple reviewers, we have provided explanations in common response to all reviewers. Based on the suggestion, in Section 4.3 of the updated paper, we discuss how our work relates to real-world tasks and limits to generalization.

Reviewer comment (Novelty):
The paper does not provide architecture-related or data-related explanation for the limitations of current PCMs. Also, it does not propose methods to address the limitations identified. Providing ideas to improve relation encoding could make the work more constructive.

Author response:
Based on our analysis, we provided some suggestions in Section 4.3 of the original paper. However, the suggestions were not clear. In the updated paper, we discuss these suggestions in Section 4.4 after pointing out the limitations. Further, we have modified the section to make the suggestion more clear.