Exploring the Role of BERT Token Representations to Explain Sentence Probing Results (2021)

Abstract: Several studies have been carried out on revealing linguistic features captured by BERT. This is usually achieved by training a diagnostic classifier on the representations obtained from different layers of BERT. The subsequent classification accuracy is then interpreted as the ability of the model in encoding the corresponding linguistic property. Despite providing insights, these studies have left out the potential role of token representations. In this paper, we provide a more in-depth analysis on the representation space of BERT in search for distinct and meaningful subspaces that can explain the reasons behind these probing results. Based on a set of probing tasks and with the help of attribution methods we show that BERT tends to encode meaningful knowledge in specific token representations (which are often ignored in standard classification setups), allowing the model to detect syntactic and semantic abnormalities, and to distinctively separate grammatical number and tense subspaces.

## introduction recent years have seen a surge of interest in pretrained language models, highlighted by extensive research around bert @xcite and its derivatives. one strand of research has focused on enhancing existing models with the primary objective of improving downstream performance on various nlp tasks @xcite @xcite . another strand analyzes the behaviour of these models with the hope of getting better insights for further developments @xcite @xcite @xcite . probing is one of the popular analysis methods, often used for investigating the encoded knowledge authors marked with a star ( ) contributed equally. 1 code is available at https://github.com/ hmohebbi/explain-probing-results in language models @xcite . this is typically carried out by training a set of diagnostic classifiers that predict a specific linguistic property based on the representations obtained from different layers. recent works in probing language models demonstrate that initial layers are responsible for encoding low-level linguistic information, such as part of speech and positional information, whereas intermediate layers are better at syntactic phenomena, such as syntactic tree depth or subject-verb agreement, while in general semantic information is spread across the entire model @xcite @xcite @xcite . despite elucidating the type of knowledge encoded in various layers, these studies do not go further to investigate the reasons behind the layer-wise behavior and the role played by token representations. analyzing the shortcomings of pre-trained language models requires a scrutiny beyond the mere performance (e.g., accuracy or f-score) in a given probing task. this is particularly important as recent studies point out that the diagnostic classifier (applied to the model's outputs) might itself play a significant role in learning nuances of the task and hence suggest evaluating probes with alternative criteria @xcite @xcite . we extend the layer-wise analysis to the token level in search for distinct and meaningful subspaces in bert's representation space that can explain the performance trends in various probing tasks. to this end, we leverage the attribution method @xcite @xcite which has recently proven effective for analytical studies in nlp @xcite @xcite @xcite . our analysis on a set of surface, syntax, and semantic probing tasks @xcite shows that bert usually encodes the knowledge required for addressing these tasks within specific token representations, particularly at higher layers. for instance, we found that sentence-ending tokens (e.g., "[sep]" and ".") are mostly responsible for carrying positional information through layers, or when the input sequence undergoes a re-ordering the alteration is captured by specific token representations, e.g., by the swapped tokens or the coordinator between swapped clauses. also, we observed that the ##s token is mainly responsible for encoding noun number and verb tense information, and that bert clearly distinguishes the two usages of the token in higher layer representations. representation subspaces. in addition to layerwise representations, subspaces that encode specific linguistic knowledge, such as syntax, have been a popular area of study. by designing a structural probe, @xcite showed that there exists a linear subspace that approximately encodes all syntactic tree distances. in a follow-up study, @xcite showed that similar syntactic subspaces exist for languages other than english in the multilingual bert and that these subspaces are shared among languages to some extent. this corroborated the finding of @xcite that multilingual bert has common subspaces across different languages that capture various linguistic knowledge. as for semantic subspaces, @xcite showed that bert places the contextualized representations of polysemous words into different regions of the embedding space, thereby capturing sense distinctions. similarly, @xcite studied bert's ability to distinguish different word senses in different contexts. using the probing approach of @xcite , they also found that there exists a linear transformation under which distances between word embeddings correspond to their sense-level relationships. our work extends these studies by revealing other types of surface, syntactic, and high-level semantic subspaces and linguistic features using a pattern-finding approach on different types of probing tasks. ## methodology our analytical study was mainly carried out on a set of sentence-level probing tasks from senteval @xcite . the benchmark consists of several single-sentence evaluation tasks. each task provides 100k instances for training and 10k for test, all balanced across target classes. we used the test set examples for our evaluation and indepth analysis. following the standard procedure for this benchmark, we trained a diagnostic classifier for each task. the classifier takes sentence representations as its input and predicts the specific property intended for the corresponding task. in what follows in this section, we first describe how sentence representations were computed in our experiments. then, we discuss our approach for measuring the attribution of individual token representations to classifier's decision. ## sentence length in this surface-level task we probe the representation of a given sentence in order to estimate its size, i.e., the number of words (not tokens) in it. to this end, we used senteval's sentlen dataset, but changed the formulation from the original classification objective to a regression one which allows a better generalization due to its fine-grained setting. the diagnostic classifier receives average-pooled representation of a sentence (cf. section 3.1) as input and outputs a continuous number as an estimate for the input length. given that the ability to encode the exact length of input sentences is not necessarily a critical feature, we do not focus on layer-wise performance and instead discuss the reason behind the performance variations across layers. to this end, we calculated the absolute saliency scores for each input token in order to find those tokens that played pivotal role while estimating sentence length. rounding the regressed estimates and comparing them with the gold labels in the test set, we can observe a significant performance drop from 0.91 accuracy in the first layer to 0.44 in the last layer (cf. appendix a.1 for details). this decay is not surprising given that the positional encodings, which are added to the input embeddings in bert and are deemed to be the main players for such a position-based task, get faded through layers @xcite . sentence ending tokens retain positional information. figure 1 shows tokens that most contributed to the probing results across different layers according to the attribution analysis. finalizing tokens (e.g. "[sep]" and ".") are the main contributors in the higher layers. we further illustrate this in figure 2 of a finalizing token with those of another frequent non-finalizing token. clearly, positioning information is lost throughout layers in bert; however, finalizing tokens partially retain this information, as visible from distinct pattern in higher layers. ## verb tense and noun number this analysis inspects bert representations for grammatical number and tense information. for this experiment we used the tense and objnum tasks foot_0 : the former checks whether the main-clause verb is labeled as present or past foot_1 , whereas the latter classifies the object according to its number, i.e., singular or plural @xcite . on both tasks, bert preserves a consistently high performance (> 0.82 accuracy) across all layers (cf. appendix a.1 for more details). articles and ending tokens (e.g., ##s and ##ed) are key playmakers. attribution analysis, illustrated in figure 3(a) , reveals that article words (e.g., "a" and "an") and the ending ##s token, which makes out-of-vocab plural words (or third person present verbs), are among the most attributed tokens in the objnum task. this shows that these tokens are mainly responsible for encoding object's number information across layers. as for the tense task, figure 3(b) shows a consistently high influence from verb ending tokens (e.g., ##ed and ##s) across layers which is in line with performance trends for this task and highlights the role of these tokens in preserving verb tense information. ## inversion abnormalities for this set of experiments, we opted for senteval's bi-gram shift and coordination inversion tasks which respectively probe model's ability in detecting syntactic and semantic abnormalities. the goal of this analysis was to to investigate if bert encodes inversion abnormality in a given sentence into specific token representations. ## control experiments the main motivation behind designing a control task in probing studies is to check whether it is the representations that encode linguistic knowledge or the diagnostic classifier itself which plays a significant role in learning nuances of the task @xcite . in this regard, most of our experiments throughout the paper (similarity curves, tsne plots, or attention-norm analysis) all rely on fixed representations and do not need any classifier or training; hence, they all serve as control experiments or sanity checks. for example, in our attention-norm analysis (which requires no training and comes from a different perspective) we arrive at the same results as our attribution analysis. computation of attribution scores based on trained diagnostic classifiers is the only part of our experiments which involves a training procedure. hence, we carried out a control study inspired by @xcite to check the consistency of attribution patterns. the intuition behind this is in line with @xcite who stated that if there is a strong regularity in the representations with respect to the labels, this can be revealed even with fewer training data points. to this end, we used only 10% of the training data to train the diagnostic classifiers and computed the attribution scores for each task. then, we computed the correlation between attribution scores for each sentence obtained by these classifiers and those obtained from the original classifiers (trained on full training data). after averaging the correlations over all examples, we report the mean and maximum statistics among all layers in ## conclusions in this paper we carried out an extensive gradientbased attribution analysis to investigate the nature of bert token representations. to our knowledge, this is the first effort to explain probing performance results from the viewpoint of token representations. we found that, while most of the positional information is diminished through layers, sentence-ending tokens are partially responsible for carrying this knowledge to higher layers in the 8 results are averaged over three runs. model. furthermore, we analyzed the grammatical number and tense information throughout the model. specifically, we observed that bert tends to encode verb tense and noun number information in the ##s token and that it can clearly distinguish the two usages of the token by separating them into distinct subspaces in the higher layers. also, we found that abnormalities can be captured by specific token representations, e.g., in two consecutive swapped tokens or a coordinator between two swapped clauses. our approach in using a simple diagnostic classifier and incorporating attribution methods provides a novel way of extracting qualitative results in probing studies. this can be seamlessly applied to various deep pre-trained models, providing a wide range of options in sentence-level tasks and from the fine-grained viewpoint of tokens. we hope this will spur future probing studies in other evaluation scenarios. future work might investigate how subspaces are evolved or transformed during fine-tuning and whether they are beneficial at inference time to various downstream tasks (e.g. syntactic abnormalities, grammatical number and tense subspaces in grammar-based tasks like cola @xcite or to check whether these behaviors are affected by different training objectives. furthermore, our token-level analysis can provide insights for enhancing model efficiency based on token importance, something we plan to pursue in future work.