A Critical Study of What Code- LLM s (Do Not) Learn (2024)

Abstract: Large Language Models trained on code corpora (code-LLMs) have demonstrated impressive performance in various coding assistance tasks. However, despite their increased size and training dataset, code-LLMs still have limitations such as suggesting codes with syntactic errors, variable misuse etc. Some studies argue that code-LLMs perform well on coding tasks because they use self-attention and hidden representations to encode relations among input tokens. However, previous works have not studied what code properties are not encoded by code-LLMs. In this paper, we conduct a fine-grained analysis of attention maps and hidden representations of code-LLMs. Our study indicates that code-LLMs only encode relations among specific subsets of input tokens. Specifically, by categorizing input tokens into syntactic tokens and identifiers, we found that models encode relations among syntactic tokens and among identifiers, but they fail to encode relations between syntactic tokens and identifiers. We also found that fine-tuned models encode these relations poorly compared to their pre-trained counterparts. Additionally, larger models with billions of parameters encode significantly less information about code than models with only a few hundred million parameters.

## introduction code-llms (cllms) are transformer models @xcite trained on a large corpus of code and natural language -programming language (nl-pl) pairs. these models are used, either in a zero-shot manner or after fine-tuning, for coding assistance tasks, including code summarization, code retrieval, code completion, code generation, and program repair @xcite . while the performance of models on benchmarks has significantly improved in the past few * work conducted while afiliated with technische universität darmstadt. prior studies have used attention analysis @xcite and probing on hidden representation @xcite to study what cllms encode. some of these studies show that models can learn the syntactic and semantic structure of code @xcite @xcite and understand code logic @xcite . however, they rely on nonsystematically validated assumptions. for example, studies on attention analysis set an arbitrary attention threshold of 0.3. the studies which probe hidden representation of code models assume a linear encoding of information. the effect of these assumptions has hitherto remained unstudied. further, these studies do not evaluate which code properties are not encoded by cllms. in this paper, we make two important contributions to advance the state of the art in the interpretability of cllms. first, we perform a systematic analysis of assumptions in previous work and show that they can lead to misleading conclusions. specifically, we examine the influence of the attention threshold and evaluation metric on attention analysis, and for probing on hidden representation, we explore whether the code relations among tokens are encoded linearly or non-linearly. to avoid several limitations of classifier and structural probing methods @xcite @xcite , we perform probing of hidden representation without any additional classifier layers or parameters. based on our observations, we make some new suggestions for experimental setup of analysis of attention maps and hidden representation. second, armed with our insights from the first analysis, we set up and perform a fine-grained analysis of attention and hidden representation of cllms at the code token level to critically examine what they learn and do not learn. previous studies examining the code comprehension ability of cllms have analyzed all input tokens together, without distinguishing between different categories of code tokens such as identifiers (e.g., function names, variables) and syntactic tokens (e.g., keywords, operators, parentheses). to investigate whether there are specific relations that cllms fail to encode, we separately analyze the syntactic-syntactic, identifier-identifier, and syntactic-identifier relations that are encoded in the self-attention values and hidden representations. there are different types of relations between code tokens, including relations in an abstract syntax tree (ast), as well as, data flow or control flow relations between code blocks. similar to @xcite , we focus primarily on syntactic relations in the ast and create a syntax graph with edges between code tokens within a motif structure (figure 6b ). but such a syntax graph does not encompass all the relations among identifiers, in particular how values flow from one variable to another. thus, we extend the study to data-flow relations and create a data flow graph (dfg) with edges among related variables following @xcite . we perform attention analysis to study whether a token pays attention to related tokens and analysis of hidden representation to study the information encoded by the model in the vector representation of a token. to study information encoded in hidden representations, we take hidden representations of pairs of tokens and evaluate if the information encoded by the model is sufficient to predict the relation between these two tokens. specifically, we evaluate with respect to predicting edges in a dfg and sibling and distance prediction in an ast. we study models with 110m to 3.7b parameters with different architectures, pre-training objectives, and training datasets foot_0 . in summary, ## related work several studies have provided some possible explaination of the working of cllms. cito et al. (2022) and rabin et al. (2023b) used input perturbation, while, liu et al. (2023) used backpropagation to find the most relevant input tokens. zhang et al. (2022a) created an aggregated attention graph and studied its application to the varmiuse task. wan et al. (2022) performed attention analysis and probing with structural probes (hewitt and manning, 2019). lópez et al. (2022) used structural probe to create binarized ast from hidden representations. probing classifiers have been used to test syntax and semantic understanding @xcite @xcite , the effect of positional embeddings @xcite , relation between self-attention and distance in ast @xcite and logic understanding @xcite . our work studies the limitations of code models in encoding code structure which has hitherto remained unexplored. our study spanning multiple transformer architectures, sizes and training objectives demonstrate a significant gap in encoding some code properties. this gap could be a possible explanation for poor performance of cllms on real-world tasks @xcite . ## experiments in this section, we elaborate on the experiments that we performed to analyze self-attention and the hidden representation of cllms. for attention analysis, we compare the self-attention of models with the motif structure in a program's ast and dfg. for hidden representations, we perform probing without classifiers using directprobe @xcite . we provide details on ast, dfg, directprobe, and motif structure in appendix b. ## conclusion in this paper, we critically examined arbitrary assumptions made in previous works on interpretability of cllms and demonstrated that these assumptions can lead to misleading conclusions. further, with improved experimental setting, we conducted an in-depth analysis of self-attention and hidden representations of cllms. the analysis revealed that cllms struggle to encode code relations between syntactic and identifier tokens. this restricts their ability to understand program flow and logic. we also observed that fine-tuned models and larger models with billions of parameters encode these relations poorly compared to smaller pre-trained models. it seems that fine-tuned and larger models rely on shortcut learning and memorized code instead of code understanding. our work contributes to designing more robust experiments to study interpretability of cllms. it also suggests that it is important to explore novel training techniques and/or architectures to enhance models' capability to encode code properties, instead of using larger models with memorization. in our future work, we aim to investigate more recent instruction-tuned models by extending this study to nl-pl alignment.