Probing Pretrained Models of Source Codes (2022)

Abstract: Deep learning models are widely used for solving challenging code processing tasks, such as code generation or code summarization. Traditionally, a specific model architecture was carefully built to solve a particular code processing task. However, recently general pretrained models such as CodeBERT or CodeT5 have been shown to outperform task-specific models in many applications. While pretrained models are known to learn complex patterns from data, they may fail to understand some properties of source code. To test diverse aspects of code understanding, we introduce a set of diagnostic probing tasks. We show that pretrained models of code indeed contain information about code syntactic structure, the notions of identifiers, and namespaces, but they may fail to recognize more complex code properties such as semantic equivalence. We also investigate how probing results are affected by using code-specific pretraining objectives, varying the model size, or finetuning.

## introduction deep learning and especially natural language processing (nlp) methods have been widely and successfully adopted to process source code. example tasks include code generation @xcite where the task is usually formulated as to produce a code of a function given the natural description; code translation @xcite where the model needs to translate from one programming language to another; and code summarization @xcite where the task is to produce natural language (nl) description for a given code snippet. deep learning is also widely used in discriminative tasks, such as automated bug search and repair @xcite . in recent years, the focus has shifted from developing task-specific models incorporating prior * now at naver labs europe knowledge about the task, to relying on general pretrained models of code such as codebert @xcite or codet5 @xcite . these models, once pretrained, can be finetuned on the downstream tasks with a little additional cost, surpassing task-specific models. while the performance of the models is high on a wide range of downstream tasks @xcite , the boundary between what the models know and where they fail remains hidden behind the complexity of the downstream tasks. the lack of interpretability of pretrained models limits their practical use. at the same time, a deeper examination of model's understanding of source code may increase developers' trust and broaden the applicability of pretrained models. in nlp, there is an established probing approach for a more fine-grained examination of the knowledge of various aspects of the language, e.g. morphology, syntax, or discourse understanding @xcite @xcite . probing usually means training a linear model on top of hidden representations of a model for various simple tasks, e.g. to predict a part-ofspeech tag, to detect whether a sentence was corrupted, or to estimate the number of objects in the main clause @xcite . probing experiments may suggest ways to improve the quality of the pretrained model or provide recommendations on how to tune the model better in applied tasks @xcite . inspired by the insights probing provided in nlp, we develop probing tasks to understand the extent to which the current state-of-the-art pretrained models capture structural and semantic properties of source code. our contributions are as follows: we release our code 1 . ## probing tasks we probe pretrained models of code using linear regression or classification trained on top of code representations extracted from each layer of each model (layers weights are not finetuned) @xcite . we develop auxiliary tasks (with synthetic data or data borrowed from other works) that test models' understanding of various properties of source code: strict syntactic structure, the notions of data flow and namespaces, naming, semantic equivalence, and readability. we consider both global tasks (predicting a property of the whole code snippet) and local tasks (predicting a property of a particular token or a group of tokens). for each task, we introduce a simple but as strong as possible baseline. figure 1 illustrates all tasks. for all classification tasks, we measure test error (1accuracy), and for "ast depth" regression task, we calculate the mean absolute error m ae(y true , y pred ) = 1 n n @xmath0. notation. pretrained models of code usually follow the standard nlp methodology: representing a 1 https://github.com/serjtroshin/probings4code code snippet as a sequence of subtokens, e. g. bytepair encoding subtokens, and pretraining the model on a large corpora of source code using masked language modeling. we denote the sequence of subtokens as s 1 , . . . , s m . let us denote t(s i ) a mapping from a subtoken s i into a corresponding code token t(s i ), e.g. for a subtoken sequence @xcite , t( get) = getstatus. for each subtoken s i , we extract the model's embedding w ℓ @xmath1, where d is the size of hidden representations. ## models in this section, we briefly describe the models to be compared. we have selected several widely used pretrained models, which vary in the model architecture, pretraining objective, model size, and training datasets. ## experiments our research questions are as follows: we used a single tesla v100 gpu for the forward pass to collect embeddings, and 4 cpu for training linear models. our total computational budget is 864 cpu hours and 20 gpu hours. ## related work probing became a universal tool in nlp for testing pretrained models' understanding or knowledge of various language aspects. a simple linear probing was used in @xcite to test whether referential knowledge is already encoded in word embeddings, while @xcite got insights into the behaviour of word embedding in terms of morphological and syntactical properties. probing tasks were developed to evaluate sentence embeddings @xcite whether they incorporate compositional, or surface (length of the sentence), syntax (tree depth, top constituent), and text semantics (e.g. tense of a sentence) knowledge. @xcite proposed more complex probing tasks, questioning the possibility to parse the whole dependency trees from the sentence embeddings using a metric learning approach. more recent studies for language models include the study of emerging capabilities of large language models @xcite . we refer to @xcite for a broad review of existing probing works in nlp. in the context of source code, @xcite made the first steps towards probing pretrained models. however, they only consider four simple tasks and tree code models, codebert, codeberta and graphcodebert. in contrast to their work, we propose a wider set of tasks, including several token-wise tasks, consider a wider a line of work investigate pretrained models for code in different directions. anonymous authors (2022) show that codebert relies on the high token overlap between query and code solving code search task rather than deeper syntax or semantic features, and @xcite shows that bert trained on code pays more attention to identifiers and separators. our work provides another view on the analysis of pretrained models of code, from the probing perspective, and complements these results. recently created big-bench benchmark @xcite ) contains a number of challenging code-related probing tasks for testing large language models capabilites, including programming synthesis and code summarization tasks, which are close to complex downstream tasks. in contrast, we aim at developing simple probing tasks targeting specific code understanding aspects. ## conclusion and discussion we presented a diagnosis tool, based on probing tasks, that can be used to estimate to which extent deep learning models capture the information about various properties of source code in their hidden representations. our results show that pretrained models of code do contain information about code syntactic structure, the notion of namespaces, data flow, code readability and natural language-based naming. however, pretrained models show limited understanding of code semantics, which means that their usefullness in applied tasks requiring semantic understanding of code may be limited. using code-specific pretraining objectives (codet5, graphcodebert) enriches the understanding of the code aspects addressed in the corresponding objective. this result may suggest practitioners to choose pretrained models which pretraining objectives are better aligned with the considered applied task. we also found that finetuning may deteriorate the model's understanding of code properties, especially in classification downstream tasks. this may suggest including code-specific objectives in finetuning, especially if multi-stage finetuning (pruksachatkun et al., 2020) is used.