Picking BERT ’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis (2020)

Abstract: As the name implies, contextualized representations of language are typically motivated by their ability to encode context. Which aspects of context are captured by such representations? We introduce an approach to address this question using Representational Similarity Analysis (RSA). As case studies, we investigate the degree to which a verb embedding encodes the verb’s subject, a pronoun embedding encodes the pronoun’s antecedent, and a full-sentence representation encodes the sentence’s head word (as determined by a dependency parse). In all cases, we show that BERT’s contextualized embeddings reflect the linguistic dependency being studied, and that BERT encodes these dependencies to a greater degree than it encodes less linguistically-salient controls. These results demonstrate the ability of our approach to adjudicate between hypotheses about which aspects of context are encoded in representations of language.

## introduction contextualized word embeddings @xcite , which are vector representations of words in context, enable neural models of language to achieve dramatic performance improvements over models whose word embeddings do not have access to context @xcite . the most obvious explanation for the success of these models is that contextualized word embeddings can incorporate contextual information, whereas other embeddings cannot. contextual information provides clues about the semantic and syntactic roles that a word plays in a sentence. for example, a verb might be understood differently when placed in different contexts: in sentence (1a), the verb charged means "ran towards," where in sentence (1b) the verb means "formally accused". there are many aspects of context that could conceivably be captured in contextualized embeddings, from linear context (e.g., what word precedes this one?) to syntactic context (e.g., what is the parent of this word in a dependency tree?). in this work, we investigate which aspects of context are captured in these embeddings. we do so by studying the embeddings' representational geometry, which is the spatial relationship between representations of stimuli. in our case, the stimuli are contextualized embeddings of words. we study this geometry by applying representational similarity analysis @xcite to the contextualized word embeddings given by bert @xcite . this allows us to ask fine-grained questions about which context words are encoded, and to what degree. 1 this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . we find that the representational geometries of bert's word and sentence embeddings reflect several linguistic dependencies. in particular, we find that bert's embeddings of verbs encode the subject of those verbs more than they encode nouns that are not arguments of those verbs. we also find that the contextualized embeddings of pronouns encode the pronouns' antecedents more than they encode other nouns. finally, we find that bert's sentence embeddings encode the main verbs of sentences more than any other content words. this is consistent with the standard assumption in dependency parsing that the main verb of a sentence is the sentence's head. these results demonstrate the ability of our approach to illuminate which aspects of context are encoded in contextualized embeddings. 2 ## background: representational geometry a representational geometry is the spatial arrangement of a set of vector representations. representational geometries are typically formed by taking the pairwise dissimilarities between those representations. for example, the representational geometry of the bert embeddings for the subjects of a set of sentences is given by the pairwise dissimilarities between the contextualized word embeddings corresponding to those subjects. when the set of representations is poorly understood, one can gain insight into it by comparing it to a set of representations that is well-understood. if the two sets have similar representational geometries, then one can infer that the sets encode similar information. representational similarity analysis (rsa) allows for comparisons between two different representational geometries. in cognitive neuroscience, this technique is used to analyze distributed activity patterns in the brain (see @xcite for a review); we use it to analyze the representations of artificial neural networks. we compare contextualized bert embeddings to representations that we construct to instantiate specific linguistic hypotheses, which we refer to as hypothesis models. these models represent specific linguistic information and abstract away from all other information @xcite . we compare the similarity among the representations of each hypothesis model to the similarity among bert's embeddings; if the representational geometry of bert's embeddings is better matched by the representational geometry of hypothesis model a than by that of hypothesis model b, we conclude that the hypothesis instantiated by a is a better description of the content of bert's embeddings. each hypothesis model represents a specific type of context word (e.g., a verb's subject), while ignoring other context words. such a hypothesis model instantiates the hypothesis that this aspect of context is the only aspect represented in bert's embeddings. of course, such extreme hypotheses are almost certainly wrong; our goal is not to find a perfect hypothesis but rather to find which of two hypotheses is closer to the truth. ## probing contextualized embeddings our approach works as follows: first, we create a corpus c of n sentences that contain the syntactic structures we wish to study (figure 1 , step 1). next, we define a reference model m ref , which consists of the representations that are being investigated (e.g., the embeddings of the main verbs from every sentence in c). then, we define two hypothesis models, m hyp 1 and m hyp 2 . these hypothesis models instantiate hypotheses about the representational geometry of the contextualized word embeddings (figure 1 , step 2 ). we then draw a sample c of n sentences from our corpus, and calculate the n × n representational geometries g of each model by applying a dissimilarity metric d to the relevant word embeddings from this sample (figure 1 , step 3 ). d(m, c) finds the dissimilarity between the representations generated by model m for each pair of sentences in the sample c. finally, we calculate the similarity s between the representational geometries of our hypothesis models and our reference model using a similarity metric sim. because the g matrices are symmetric, sim only operates on the upper triangle of each g matrix. we then repeat the process on m samples from our corpus in order to create two m-length vectors of representational similarities, s hyp 1 and s hyp 2 (figure 1 , step 4 ). finally, we apply a nonparametric sign test to the difference of these vectors, s hyp 1 -s hyp 2 , to test whether there is a consistent difference between measurements of s hyp 1 and s hyp 2 . in the following section, we will walk through this approach with a concrete example. ## experiment 1: subject-sensitivity of verb embeddings most verbs in english require arguments, such as subjects and direct objects. here we focus on subjects, which are required by nearly all verbs. specifically, we consider whether the contextualized word embedding of the main verb encodes the subject of a sentence to a greater degree than it encodes other nouns in the sentence that are not arguments of the verb. in particular, we study the verbs is and are. ## experiment 2: pronoun coreference like the meanings of verbs, the meanings of pronouns can also be affected by context. pronouns typically refer to some contextually-understood noun, which is called its antecedent. for example, in i love new york -it's my favorite city, it refers to new york; whereas in the death star is a threat to the galaxy, so it must be destroyed, it refers to the death star. here we investigate whether the contextualized representations of pronouns encode information about the pronouns' antecedents. we focus on two types of pronouns, reflexives and pronominals. reflexives must refer to a locallyoccurring (i.e., in the same clause) noun. pronominals must not refer to a locally-occurring noun. sentence (5a) contains a reflexive (himself ), which refers to the noun politician, while sentence (5b) contains a pronominal (him), referring to the noun person. note that, in both cases, the pronoun cannot refer to the other noun (underlined). we study both types of pronoun separately. we thus generate one corpus containing sentences of the same form as example (5a), and another containing sentences of the same form as example (5b), using a probabilistic context-free grammar. we exclude any sentence where both nouns are the same. this yields 1,828 valid sentences in our reflexive corpus, and 1,826 valid sentences in our pronominal corpus. for both corpora, we consider the same reference model: the set of bert embeddings of the pronouns. we also consider two hypothesis models: the antecedent model concatenates the glove embedding of the pronoun with the glove embedding of its antecedent for every sentence. note that the antecedent is the first noun in the sentence for the pronominal corpus, and the second noun in the sentence for the reflexive corpus ( bolded alex gave sam a gift. antecedent noun is the first noun in the sentence for the reflexive corpus, and the second noun in the sentence for the pronominal corpus (underlined in example (5a) and example ( 5b )). we also include a null hypothesis model, which concatenates the glove embedding of the pronoun with the embedding of a random noun from our grammar that does not appear in the sentence. these hypothesis models allow us to test the hypothesis that the bert embeddings of pronouns encode their antecedents more than any other noun in the sentence. results: we find that, for both corpora, the representational geometry of the set of bert embeddings of the pronouns is significantly more similar to the hypothesis model that represents antecedent nouns than the hypothesis model that represents non-antecedent nouns (see ## experiment 3: heads of sentences our approach can be applied to any type of representation that incorporates information from multiple words. so far we have applied it to contextualized word representations, but it can also be applied to full-sentence representations; here we give an example of such a usage inspired by dependency parsing. in standard approaches to dependency parsing @xcite , the main verb of a sentence acts as the head of the entire sentence, as in figure 4 . we study whether verbs exhibit the same primacy in bert embeddings as they do in dependency parses. to do so, we use the embedding of the [cls] token as an embedding of the full sentence, as is standard when a sentence embedding is required from bert. we create four corpora, each containing a different type of sentence structure. these corpora are denoted the intransitive corpus, intransitive + adjective corpus, transitive corpus, corpus example intransitive the painter swims. intransitive + adjective the happy politician talks. ## comparisons to other approaches one popular approach for analyzing neural network models is the behavioral approach, in which a model's performance is evaluated on some challenge set designed to highlight particular linguistic phenomena. these methods have been used to determine whether various language models track dependencies based on syntactic number @xcite @xcite . additionally, natural language inference tasks directly test a model's ability to infer semantic relationships @xcite . for more examples, see @xcite . behavioral methods are well-suited for holistic questions about a model's overall handling of language, but here our goal is to analyze specific internal representations independently of behavior. for this question, the behavioral approach is less well-suited because it can only give an indirect window into the structure of the representational space; analyzing the internal representations themselves is much more direct. another popular approach, the diagnostic model approach, permits analyses that directly investigate internal representations. diagnostic models are simple (usually linear) classification or regression models that take in activation patterns and predict some information of interest. if the diagnostic model performs well, then the activation pattern is typically said to encode this information. prior work has used this method to determine that bert's representations are sensitive to the hierarchical structure of language @xcite , contain information relevant to a variety of tagging tasks @xcite , contain semantic information @xcite , and contain much other linguistically-useful information @xcite . these usages of diagnostic models are generally focused on testing whether an embedding does or does not represent certain information, but we instead focus on the degree to which the embedding represents information from particular words, and standard diagnostic-model approaches make it difficult to obtain relative results of this sort, because it is possible that information from all words in the sentence is represented in the embedding to some degree. this claim is supported by recent work, which finds that several salient linguistic features can be recovered from every word in a sentence using diagnostic classifiers @xcite ). thus, it seems that a diagnostic classifier approach may be insufficient for answering the questions that we address here. indeed, we attempted to devise a diagnostic model approach and applied it to our pronoun coreference and verb subject-sensitivity studies, but found little success (see appendix a). a modification to the diagnostic model approach which makes this approach more amenable to illuminating relative strengths of encoding is minimum description length (mdl) probing, introduced in the concurrent work of @xcite . mdl probing characterizes how regularly specific information is encoded in vectors, while our approach investigates the representational geometry of a set of encodings. these concepts of regularity and representational geometry are likely related, but we leave for future work an investigation of the precise relationship between them. one final approach that has been used to study whether linguistic dependencies are captured in bert is to analyze whether specific attention heads track particular dependency relationships @xcite . though this approach enables analysis of the same linguistic phenomena that we study, it does not directly demonstrate whether or how this information is encoded in bert's vector representations. instead, it illustrates which inputs are most salient in generating these vector representations, without revealing what information from those inputs is encoded in the final representation. in addition, because bert uses many attention heads, the behavior of any one attention head gives only a partial picture of the inputs for a given vector. ## related work other applications of rsa in nlp: previous work has also explored the application of representational similarity analysis to neural networks. indeed, one of the first applications of this type of analysis was performed on artificial neural systems @xcite . @xcite use rsa to characterize how the representational geometries of various models change when given different amounts of context. other work has used rsa to compare the semantic representations of cnn object detection models with that of word vectors @xcite , to compare the representations of utterances in a spoken-word encoder to the representations of those same words in the text or visual domain @xcite , and to compare the representational spaces of two agents in an emergent language game @xcite . @xcite also employ rsa to study the correspondence between the representational geometry of various language models (including bert) and a hypothesis model based on gold syntax trees. they introduce a new technique, rsa regress , which merges rsa with a diagnostic model approach. the present work is complementary to their study. whereas their study compares different neural models against one hypothesis model, our study compares multiple hypothesis models against a single neural model, in order to adjudicate between specific linguistic hypotheses. geometric analyses of nlp systems: though we study the representational geometry of contextualized word embeddings, other work has been done to characterize the geometry of the network activations themselves. these are distinct concepts, as our current work can be thought of as analyzing the secondorder geometry (comparing the relationships between representations of one model to the relationships between representations of another), while this prior work is analyzing first-order geometry (comparing the representations of one model to the representations of another). this first-order approach has also been used to analyze models' linguistic capabilities @xcite @xcite . most directly related, @xcite uses a first-order approach to analyze bert's representation of subject-verb agreement and pronoun coreference. ## conclusion and future work we have introduced a framework for using representational similarity analysis to adjudicate between hypotheses about the representational geometry of a neural network's embeddings. we then applied this procedure to bert embeddings, and demonstrated that they are sensitive to linguistic dependencies. in particular, we showed that bert's embeddings of pronouns encode the pronouns' antecedents more than they encode other nouns, that the embeddings of verbs encode the verbs' subjects more than they encode non-argument nouns, and that bert's sentence embeddings most saliently encode the sentences' main verbs, as predicted by standard dependency frameworks. not only do these studies reveal a sensitivity to linguistic dependencies, two of them (the subject-sensitivity and pronoun-coreference studies) demonstrate that these dependencies are more salient than relationships between words at the surface level. this framework can enable the investigation of many linguistically-motivated questions. as long as hypothesis models can be defined to instantiate the relevant linguistic hypotheses, this framework allows researchers to study which of the hypotheses is closer to the truth. additionally, future work can focus on making more complete hypothesis models. our studies only required us to account for the differences between the representational similarity of different hypothesis models, but not the magnitude of that similarity. in future work, we plan to focus on creating more complex hypothesis models that exhibit greater absolute representational similarity to the reference model.