Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words (2020)

Abstract: Although models using contextual word embeddings have achieved state-of-the-art results on a host of NLP tasks, little is known about exactly what information these embeddings encode about the context words that they are understood to reflect. To address this question, we introduce a suite of probing tasks that enable fine-grained testing of contextual embeddings for encoding of information about surrounding words. We apply these tasks to examine the popular BERT, ELMo and GPT contextual encoders, and find that each of our tested information types is indeed encoded as contextual information across tokens, often with near-perfect recoverability—but the encoders vary in which features they distribute to which tokens, how nuanced their distributions are, and how robust the encoding of each feature is to distance. We discuss implications of these results for how different types of models break down and prioritize word-level context information when constructing token embeddings.

## introduction the field of natural language processing has recently seen impressive performance gains associated with the use of "contextual word embeddings": high-dimensional vectors that have access to information from the contexts of the words they represent. models that use these contextual embeddings achieve state-of-the-art performance on a variety of natural language processing tasks, from questionanswering to natural language inference. as of writing, nearly all of the models on the superglue leaderboard @xcite use contextual embeddings in their architectures, most notably models building on the bert @xcite and transformer xl @xcite models. despite the clear power afforded by incorporating context into word embeddings, little is known about what information these contextual embeddings actually encode about the words around them. in a sentence like "the lawyer questioned the judge", does the contextual representation for questioned reflect properties of the subject lawyer? of the object judge? what determines the information that a contextual embedding absorbs about its surrounding words? in this paper, we address these questions by designing and implementing a suite of probing tasks, to test contextual embeddings for information about syntactic and semantic features of words in their contexts. we use controlled sentences of fixed structure, allowing us to probe for information associated with word categories, and to avoid confounds with particular vocabulary items. we then apply these tests to examine the distribution of contextual information across token representations produced by contextual encoders bert @xcite , elmo @xcite , and gpt @xcite . the contributions of this paper are twofold. first, we introduce a suite of novel probing tasks for testing how encoders distribute contextual information across sentence tokens. all datasets and code are available for follow-up testing. 1 second, we use these tests to shed light on the distribution of context information in state-of-the-art encoders bert, elmo and gpt. we find that these models encode each of our tested word features richly across sentence tokens, often with perfect or near-perfect recoverability, but the details of how the models distribute this information vary across encoders. in particular, bidirectional models show more nuance in information selectivity, while the deeper transformer models show more robustness to distance. follow-up tests suggest that the effects cannot be chalked up to proximity, and that general word features are encoded more robustly than word identity. 1 probing datasets and code available at https://github.com/jklafka/context-probes . the lawyer questioned the judge how well can we extract information about the subject noun (lawyer)? what if we probe the object noun (judge) or determiners (the)? we develop tasks to probe representations of each word for various types of information about the other words of the sentence, allowing us to examine with fine granularity how contextual encoders distribute information about surrounding words. we complete this investigation for each word in a set of fixedlength sentences of pre-determined form, which allows us to characterize behaviors based on word categories (e.g., subjects versus verbs). using this approach, we can examine how the distribution of context information is impacted by a) the type of information being encoded, and b) the properties of the word that the embedding corresponds to. ## related work much work has been done on analyzing the information captured by sentence encoders and language models in general. classification-based probing tasks have been used to analyze the contents of sentence embeddings @xcite @xcite , finding that these embeddings encode a variety of information about sentence structure, content, length, etc., though more tightly-controlled tasks suggest weaknesses in capturing basic sentence meaning @xcite . our work uses the same classificationbased probing methodology, but focuses on probing token-level embeddings for context information. other work has analyzed linguistic capacities of language models by examining output probabilities in context, emulating methods for studying human language processing. much of this work has studied sensitivity to syntactic dependencies in recurrent neural network language models (e.g. @xcite @xcite @xcite . using similar methods to test syntactic awareness in bert, @xcite finds the model to perform almost at ceiling on syntactic tests. testing bert's outputs on a range of semantic, syntactic and pragmatic information, ettinger (2020) finds strong sensitivity to syntax, but clear limitations in areas of semantics and pragmatic/commonsense reasoning. we complement this work with a direct focus on the contextual token representations learned by models pre-trained on language modeling, examining the syntactic and semantic information that these embeddings capture about surrounding words. most directly related to the present work are studies using probing and other methods to analyze information in contextual token embeddings. some of this research (e.g. @xcite finds that bert encodes more local, syntactic information at lower layers and more global, semantic information at higher layers. @xcite find that encoders differ in encoding strength for semantic features but all encode these features strongly where possible. hewitt and manning (2019) provide evidence that contextual encoders capture sentence-level hierarchical syntactic structures in their representations. other work @xcite finds that contextual word encoders struggle to learn fine-grained linguistic information in a variety of contexts. these papers have focused primarily on studying the ability of contextual embeddings to capture information about the full sentence, or about phrases or dependencies of which those contextual embeddings are a part. we focus on mapping the precise distribution of context information across token embeddings, with a systematic, finegrained investigation of the information that each token encodes about each of its surrounding tokens. ## probing for contextual information for each of our probing tasks, we test for a particular information type, formulated as a query about a particular target word in the sentence-for instance, "what is the animacy of the subject?" or "what is the tense of the verb?". we then apply these queries to probe the embeddings for each word of the sentence in turn-we call this the probed word. for example, a test with a probed word of verb, a target word of subject noun, and an information type of animacy would ask the question: "what does the embedding of the verb tell us about the animacy of the subject noun?" we implement each test as a binary classification task (e.g., "animate" vs "inanimate"), and train and test a multi-layer perceptron task example label (subject) number the lawyer betrayed the judge. singular the lawyers betrayed the judge. ## experiments we apply our probing tasks to test for the distribution of contextual information across tokens in three prominent contextual encoders: bert base @xcite , elmo @xcite , and gpt @xcite . bert base is a bidirectional transformer architecture of 12 layers, trained on a novel masked language modeling task of predicting randomly masked tokens using left and right context, as well as a next-sentence prediction task. we probe representations from the model's final layer, based on results suggesting that bert's later layers contain more semantic and abstract information (e.g. @xcite . elmo is composed of stacked bidirectional lstms, trained by jointly optimizing backwards and forwards language modeling objectives. we use the original version of elmo with two representation layers, and we probe representations from the second layer, which has also been found to encode more abstract and semantic information @xcite . gpt is a unidirectional left-to-right 12-layer transformer, also trained on language modeling. consistent with elmo and bert, we probe representations from gpt's final layer. 3 we test the pre-trained versions of these models without fine-tuning, to examine their general-purpose encoding capacities, in line with @xcite . we use these models to embed each of our fiveword sentences, producing contextualized representations for each token. then for each probing task (e.g., subject animacy, verb tense) we train and test classifiers on the embeddings for a single probed word category (e.g., object noun) at a time. we use several classifier architectures in our probing tasks, in order to explore the impact of classifier complexity on extraction of our target information types. we use a multilayer perceptron classifier with a single hidden layer of 1024 units, as well as a smaller classifier with three layers of 20 units each, and a larger classifier with three layers of 1024 units each. we use the relevant contextual or non-contextual token representations as input for classification. the largest inputs we supply to the classifiers are contextual embeddings with dimension 1024, from elmo.we use the relevant contextual or non-contextual token representations as input to the classifiers. finding similar results across classifier architectures, we follow precedent in the literature @xcite and present results only for our classifier with a single hidden layer. to quantify variance across runs, we repeat this process 50 times for each probed word category on each task. 4 as a sanity-check baseline, we also test noncontextual glove embeddings @xcite on each of our tasks, to establish how well each information type is captured by the noncontextual representation for the relevant word (e.g., does the glove embedding for waiters encode the information that waiters is plural? masculine?). we also want to confirm that none of these tasks can be performed by non-contextual embeddings for any of the other words of the sentence, to ensure that the information being tested for is truly contextual. we use 300-dimensional glove embeddings, which prove generally adequate to encode all of the targeted word information. figures 1 2 3 show the results for tasks with subject noun, object noun, and verb target words, respectively (note that although the plots include tokens from an example sentence for purposes of clarity, these are results across all test sentences). each cluster of adjacent bars of the same shade repre-sents the three different tested information types, with left-to-right order of number-gender-animacy for noun target words, and tense-dynamic-causative for verb target words. ## distance manipulation tasks setup because our sentences follow a fixed structure for category-specific probing, it is possible that differences in encoding from word to word are an effect of linear distance rather than the syntactic/semantic relationships between the words. we perform a follow-up analysis inspired by a task in @xcite , in which the authors investigate the effect of distance from the target word as a factor in how richly recurrent neural networks encode syntactic information. for all of our tasks, we introduce a manipulation to change linear distances between our target and probed words, by splicing relative clauses after the subject and adjectives before the object. for example: the lawyer found the judge. the lawyer who was hungry found the angry and competent judge. for reasons of space, we display only subject task results, in figure 4 . all results may be found in our github repository linked in footnote 1. ## word identity tasks setup we aim to show whether the encoders incorporate only more coarse-grained linguistic information in their embeddings, or if encoding is fine-grained enough to memorize the embedding patterns for specific context word identities. we use a variation of the word content task from @xcite and @xcite . the goal of the original word content task is to determine whether a sentence vector representation contains a given word. we adapt this task to test the extent to which contextual embeddings can identify a neighboring word at a given position. we formulate our word identity tasks as "what is the identity of the subject" or "what is the identity of the verb", figure 5: word identity task: labeling identity of subject noun. vertical ranges show 95% confidence intervals computed with non-parametric bootstrap. for identity classification, we use a softmax kway classification task, similar to the word content task in @xcite . the classifier for this task must choose which of the k output words is in the target position of the sentence. in pre-testing, we found best overall performance with a 30-way classification, for which we present the results here. smaller and larger k produce similar patterns of results, but performance overall decreases. ## discussion the results presented here shed light on how different contextual encoders distribute information across token embeddings of a sentence. while we cannot draw strong conclusions about causal relations between model properties and the observed patterns, we can make broad connections between the two to inform future investigations. overall, the deeper, transformer-based architectures of bert and gpt do not produce dramatic differences in distribution of information relative to the shallower elmo model-the main difference observed with elmo's shallower recurrent architecture is a bit of a drop in information (particularly number and word identity) over longer distances, where bert and gpt retain strong encoding. this is not necessarily surprising, given the potential of the self-attention mechanism to capture long-distance connections-it is perhaps more surprising that elmo shows so little difference overall. these patterns suggest that deeper transformer models may not be critical for encoding and distributing these types of context information, except perhaps over substantial distances. bert and elmo, the models that use bidirectional context, generally pattern more similarly to each other than to gpt, particularly in strongly encoding number and animacy over gender, and encoding number strongest overall for nouns; gpt shows more uniformity in encoding noun information (at least from the subject noun). this pattern suggests that using bidirectional versus unidirectional context has more impact on distribution of context information than does depth or architecture type. gpt's poor encoding of object information relative to subject and verb information further suggests that the left-to-right architecture may prioritize earlier information over later information. as for the two bidirectional models, what bert's particular properties seem to give it over elmo, beyond more robustness to distance, is slightly different selectivity-dropping subject gender information earlier than elmo does, while keeping object animacy information at a longer distance, and dropping verb tense information a bit more. given bert's generally stronger performance on downstream tasks, this suggests that bert's masked language modeling setup, in tandem with its greater capacity to handle longer distances, allows for a more nuanced picture of how bidirectional context information should be distributed across tokens for optimal predictive power. ## conclusion in this paper we have begun to tackle a key question in our understanding of the contextual embeddings on which most current state-of-the-art nlp models are founded: what is it that contextual embeddings pick up about the words in their contexts? we have introduced a novel probing approach and a suite of tasks through which we have performed systematic, fine-grained probing of contextual token embeddings for information about features of their surrounding words. we apply these tests to examine the distribution of contextual information across sentence tokens for popular contextual encoders bert, elmo, and gpt. we find that each of the tested word features can be encoded in contextual embeddings for other words of the sentence, often with perfect or nearperfect recoverability. however, we see substantial variation across encoders in how robustly each information type is distributed to which tokens. distance manipulations indicate that the observed rich contextual encoding is not an artifact of proximity between words, and probing for information about context word identities suggests a weaker encoding of identity information than of more abstract word feature information. bidirectional context appears to impact distribution patterns more than depth or architecture, though the transformer models show more robustness to distance. overall, these results help to clarify the patterns of distribution of context information within contextual embeddingsfuture work can further clarify the impact of more diverse syntactic relations between words, and of additional types of word features. we make all datasets and code available for additional testing.