Linguistic Profiling of a Neural Language Model (2020)

Abstract: In this paper we investigate the linguistic knowledge learned by a Neural Language Model (NLM) before and after a fine-tuning process and how this knowledge affects its predictions during several classification problems. We use a wide set of probing tasks, each of which corresponds to a distinct sentence-level feature extracted from different levels of linguistic annotation. We show that BERT is able to encode a wide range of linguistic characteristics, but it tends to lose this information when trained on specific downstream tasks. We also find that BERT’s capacity to encode different kind of linguistic properties has a positive influence on its predictions: the more it stores readable linguistic information of a sentence, the higher will be its capacity of predicting the expected label assigned to that sentence.

## introduction neural language models (nlms) have become a central component in nlp systems over the last few years, showing outstanding performance and improving the state-of-the-art on many tasks @xcite @xcite . however, the introduction of such systems has come at the cost of interpretability and, consequently, at the cost of obtaining meaningful explanations when automated decisions take place. recent work has begun to study these models in order to understand whether they encode linguistic phenomena even without being explicitly designed to learn such properties @xcite @xcite . much of this work focused on the analysis and interpretation of attention mechanisms @xcite @xcite and on the definition of probing models trained to predict simple linguistic properties from unsupervised representations. probing models trained on different contextual representations provided evidences that such models are able to capture a wide range of linguistic phenomena @xcite @xcite and even to organize this information in a hierarchical manner @xcite @xcite . however, the way in which this knowledge affects the decisions they make when solving specific downstream tasks has been less studied. in this paper, we extended prior work by studying the linguistic properties encoded by one of the most prominent nlm, bert @xcite , and how these properties affect its predictions when solving a specific downstream task. we defined three research questions aimed at understanding: (i) what kind of linguistic properties are already encoded in a pre-trained version of bert and where across its 12 layers; (ii) how the knowledge of these properties is modified after a fine-tuning process; (iii) whether this implicit knowledge affects the ability of the model to solve a specific downstream task, i.e. native language identification (nli). to tackle the first two questions, we adopted an approach inspired to the 'linguistic profiling' methodology put forth by @xcite , which assumes that wide counts of linguistic features automatically extracted from parsed corpora allow modeling a specific language variety and detecting how it changes with respect to other varieties, e.g. complex vs simple language, female vs male-authored texts, texts written in the same l2 language by authors with different l1 languages. particularly relevant for our study, is that multi-level linguistic features have been shown to have a highly predictive role in tracking the evolution of learners' linguistic competence across time and developmental levels, both in first and second language acquisition scenarios @xcite . given the strong informative power of these features to encode a variety of language phenomena across stages of acquisition, we assume that they can be also helpful to dig into the issues of interpretability of nlms. in particular, we would like to investigate whether features successfully exploited to model the evolution of language competence can be similarly helpful in profiling how the implicit linguistic knowledge of a nlm changes across layers and before and after tuning on a specific downstream task. we chose the nli task, i.e. the task of automatically classifying the l1 of a writer based on his/her language production in a learned language @xcite . as shown by @xcite , linguistic features play a very important role when nli is tackled as a sentence-classification task rather than as a traditional document-classification task. this is the reason why we considered the sentence-level nli classification as a task particularly suitable for probing the nlm linguistic knowledge. finally, we investigated whether and which linguistic information encoded by bert is involved in discriminating the sentences correctly or incorrectly classified by the fine-tuned models. to this end, we tried to understand if the linguistic knowledge that the model has of a sentence affects the ability to solve a specific downstream task involving that sentence. contributions in this paper: (i) we carried out an in-depth linguistic profiling of bert's internal representations (ii) we showed that contextualized representations tend to lose their precision in encoding a wide range of linguistic properties after a fine-tuning process; (iii) we showed that the linguistic knowledge stored in the contextualized representations of bert positively affects its ability to solve nli downstream tasks: the more bert stores information about these features, the higher will be its capacity of predicting the correct label. ## related work in the last few years, several methods have been devised to obtain meaningful explanations regarding the linguistic information encoded in nlms @xcite . they range from techniques to examine the activations of individual neurons @xcite @xcite to more domain specific approaches, such as interpreting attention mechanisms @xcite @xcite , studying correlations between representations @xcite or designing specific probing tasks that a model can solve only if it captures a precise linguistic phenomenon using the contextual word/sentence embeddings of a pre-trained model as training features @xcite @xcite . these latter studies demonstrated that nlms are able to encode a variety of language properties in a hierarchical manner @xcite @xcite and even to support the extraction of dependency parse trees @xcite . @xcite investigated the representations learned at different layers of bert, showing that lower layer representations are usually better for capturing surface features, while embeddings from higher layers are better for syntactic and semantic properties. using a suite of probing tasks, @xcite found that the linguistic knowledge encoded by bert through its 12/24 layers follows the traditional nlp pipeline: pos tagging, parsing, ner, semantic roles and then coreference. @xcite , instead, quantified differences in the transferability of individual layers between different models, showing that higher layers of rnns (elmo) are more task-specific (less general), while transformer layers (bert) do not exhibit this increase in task-specificity. ## our approach to probe the linguistic knowledge encoded by bert and understand how it affects its predictions in several classification problems, we relied on a suite of 68 probing tasks, each of which corresponds to a distinct feature capturing lexical, morpho-syntactic and syntactic properties of a sentence. specifically, we defined three sets of experiments. the first consisted in probing the linguistic information learned by a pre-trained version of bert (bert-base, cased) using gold sentences annotated according to the universal dependencies (ud) framework @xcite . in particular, we defined a probing model that uses bert contextual representations for each sentence of the dataset and predicts the actual value of a given linguistic feature across the internal layers. the second set of experiments consisted in investigating variations in the encoded linguistic information between the pre-trained model and 10 different fine-tuned ones obtained training bert on as many native language identification (nli) binary tasks. to do so, we performed again all probing tasks using the 10 fine-tuned models. for the last set of experiments, we investigated how the linguistic competence contained in the models affects the ability of bert to solve the nli downstream tasks. ## profiling bert our first experiments investigated what kind of linguistic phenomena are encoded in a pre-trained version of bert. to this end, for each of the 12 layers of the model (from input layer -12 to output layer -1), we firstly represented each sentence in the ud dataset using the corresponding sentence embeddings according to the criterion defined in sec. 3.3. we then performed for each sentence representation our set of 68 probing tasks using the linearsvr model. since most of our probing features are strongly correlated with sentence length, we compared the probing model results with the ones obtained with a baseline computed by measuring the spearman's rank correlation coefficient (ρ) between the length of the ud dataset sentences and the corresponding probing values. the evaluation is performed with a 5fold cross validation and using spearman correlation (ρ) between predicted and gold labels as evaluation metric. figure 2 : layerwise ρ scores for the 68 linguistic features. absolute baseline scores are reported in column b. as a first analysis, we probed bert's linguistic competence with respect to the 9 groups of probing features. we then focused on how bert's linguistic competence changes across layers. these results are reported in figure 1 , where we see that the average layerwise ρ scores are lower in the last layers both for all distinct groups and for all features together. as suggested by @xcite , this could be due to the fact that the representations that are better-suited for language modeling (output layer) are also those that exhibit worse probing task performance, indicating that transformer layers trade off between encoding general and probed features. however, there are differences between the considered groups: competences about raw texts features (rawtext) and the distribution of pos are lost in the very first layers (by layer -10), while the knowledge about the order of subject/object with respect to the verb, the use of subordination, as well as features related to verbal predicate structure is acquired in the middle layers. interestingly, if we consider how the knowledge of each feature changes across layers (figure 2 ), we observe that not all features belonging to the same group have an homogeneous behaviour. this is for example the case of the two features included in the rawtext group: word length (char per tok) achieves quite lower scores across all layers with respect to the sent length feature. similarly, the knowledge about pos differs when we consider more granular distinctions. for instance, within the broad categories of verbs and nouns, worse predictions are obtained by sub-specific classes of verbs based on tense, person and mood features (see especially past participle, xpos dist vbn), and by inflected nouns both singular and plural ( nn, nns). within the broad set of features extracted from syntactic annotation, we also see that different scores are reported for features referring e.g. to types of dependency relations: those linking a functional pos to its head (e.g. dep dist case, dep dist cc, dep dist conj, dep dist det) are kor tel hin jpn chi tur ara ger fre spa baseline 59.05 51.32 54.09 56.27 55.68 55.66 52.92 59.29 56.03 52.61 bert 85.74 85.18 84.75 84.19 82.78 79.29 76.38 72.78 72.50 70.03 note that, for each task, sentences of the 10 languages are paired with the italian ones (e.g. @xmath0. ## the impact of fine-tuning on linguistic knowledge once we have probed the linguistic knowledge encoded by bert across its layers, we investigated how it changes after a fine-tuning process. to do so, we started with the same pre-trained version of the model used in the previous experiment and performed a fine-tuning process for each of the 10 subsets built from the original nli corpus (sec. 3.1). we decided to use 50% of each nli subset for training (40% and 10% for training and development set) and the remaining 50% for testing the accuracy of the newly generated models. after fine-tuning the model on nli, we performed again the suite of probing tasks on the ud dataset using the 10 newly generated models and following the same approach discussed in section 4. figure 4 reports layerwise mean ρ correlation values for all probing tasks obtained with bert-base and the other fine-tuned models. it can be noticed that the representations learned by the nlm tend to lose their precision in encoding our set of linguistic features after the fine-tuning process. this is particularly noticeable at higher layers and it possibly suggests that the model is storing task-specific information at the expense of its ability to encode general knowledge about the language. again, this is particularly evident for the models fine-tuned on the classification of language pairs belonging to the same family, spa-ita above all. to study which phenomena are mainly involved in this loss, we computed the differences between the probing tasks results obtained before and after the fine-tuning process. we focused in particular on the scores obtained on the output layer representations (layer -1), since it is the most task-specific @xcite . for each subset, figure 5 reports the difference between the score of each linguistic feature obtained with the pre-trained model and the fine-tuned one. not surprisingly, the loss of linguistic knowledge reflects the typological trend observed for overall classification performance. in fact, when the task is to distinguish italian vs german, french and spanish l1, bert loses much of its encoded knowledge for almost all the considered features. this is particularly evident for the morpho-syntactic features (i.e. distribution of upos dist and xpos dist) and for features related to lexical variety (i.e. ttr form, ttr lemma). it seems that for typologically similar languages bert needs more task-specific knowledge mostly encoded at the level of morpho-syntactic information rather than the structural level. on the contrary, the drop is less pronounced and in most cases not significant for models fine-tuned on the classification of more distant languages (e.g. models fine-tuned on kor-ita or tur-ita). in this case, the quite stable performance on the probing tasks may suggest that those features were still useful to perform the downstream task. interestingly, the class of features that decreases significantly in all models are those encoding the knowledge about the tense of verbs. this is particularly the case of the third-person singular verbs in the present tense (xpos dist vbz) and of verbs in the past tense (xpos dist vbd). a possible explanation could be related to the prompts of essays, which are the same across the nli dataset. thus, the textual genre could have favored a quite homogeneous use of verbal morphology features by students of all l1s. this makes this class of features less useful for the identification of native languages. ## are linguistic features useful for bert's predictions? as a last research question we investigated whether the implicit linguistic knowledge affects bert's predictions when solving the nli downstream task. to answer this question we have split each nli subset into two groups, i.e. sentences correctly classified according to the l1 and those incorrectly classified. for the two groups of each nli subset, we performed the probing tasks using the pre-trained bert-base and the specific nli fine-tuned model. for each sentence of the two groups, we calculated the variation between the actual and predicted feature value obtaining two lists of absolute errors. we used the wilcoxon rank-sum test to verify whether the two lists were selected from samples with the same distribution. as a general remark, we observed that much more than half of features vary in a statistically significant way between correctly and incorrectly classified sentences. this suggests that bert's linguistic competence on the two groups of sentences is very different. to deepen the analysis of this difference, we calculated the accuracy achieved by bert in terms of mean square error (mse) only for the set of features varying in a significant way. figure 6 reports the percentage of features for which the mse of the sentences correctly classified (mse pos) is lower than that of the incorrectly ones (mse neg). this percentage is significantly higher, thus showing that bert's capacity to encode different kind of linguistic information could have an influence on its predictions: the more bert stores readable linguistic information into the representations it creates, the higher will be its capacity of predicting the correct l1. moreover, we noticed that this is true also (and especially) using the pre-trained model. in other words, this result suggests that the evaluation of the linguistic knowledge encoded in a pre-trained version of bert on a specific input sequence could be an insightful indicator of its ability in analyzing that sentence with respect to a downstream task. interestingly, if we analyze the average length of correct and incorrect classified sentences, the correct ones are much more longer than the others for all tasks (from 3 tokens more for spa-ita to 9 for tel-ita). this is quite expected for the nli task, since a higher number of linguistic events possibly occurring in longer sentences are needed to classify the l1 of a sentence @xcite . at the same time, longer sentences make more complex the probing tasks because the output space is larger for almost all them. this is an additional evidence that bert's linguistic knowledge is not strictly related to sentence complexity, but rather to the model's ability to solve a specific downstream task. to confirm this hypothesis and verify whether such tendency does not only depend on sentence length, we trained another linearsvr that takes as input the sentence length and predict our probing tasks according to correctly or incorrectly classified nli sentences. ## conclusion in this paper we studied what kind of linguistic properties are stored in the internal representations learned by bert before and after a fine-tuning process and how this implicit knowledge correlates with the model predictions when it is trained on a specific downstream task. using a suite of 68 probing tasks, we showed that the pre-trained version of bert encodes a wide range of linguistic phenomena across its 12 layers, but the order in which probing features are stored in the internal representations does not necessarily reflect the traditional division with respect to the linguistic annotation levels. we also found that bert tends to lose its precision in encoding our set of probing features after the finetuning process, probably because it is storing more task-related information for solving nli. finally, we showed that the implicit linguistic knowledge encoded by bert positively affects its ability to solve the tested downstream tasks.