Context Analysis for Pre-trained Masked Language Models (2020)

Abstract: Pre-trained language models that learn contextualized word representations from a large un-annotated corpus have become a standard component for many state-of-the-art NLP systems. Despite their successful applications in various downstream NLP tasks, the extent of contextual impact on the word representation has not been explored. In this paper, we present a detailed analysis of contextual impact in Transformer- and BiLSTM-based masked language models. We follow two different approaches to evaluate the impact of context: a masking based approach that is architecture agnostic, and a gradient based approach that requires back-propagation through networks. The findings suggest significant differences on the contextual impact between the two model architectures. Through further breakdown of analysis by syntactic categories, we find the contextual impact in Transformer-based MLM aligns well with linguistic intuition. We further explore the Transformer attention pruning based on our findings in contextual analysis.

## introduction pre-trained masked language models (mlm) such as bert @xcite and albert @xcite have set state-of-the-art performance on a broad range of nlp tasks. the success is often attributed to their ability to capture complex syntactic and semantic characteristics of word use across diverse linguistic contexts @xcite . yet, how these pre-trained mlms make use of the context remains largely unanswered. recent studies have started to inspect the linguistic knowledge learned by pre-trained lms such as word sense @xcite , syntactic parse trees @xcite , and semantic relations @xcite . others directly analyze model's intermediate representations and attention weights to understand how they work @xcite . while previous works either assume access to model's internal states or take advantage of model's special structures such as self-attention maps, these analysis are difficult to generalize as the architectures evolve. in this paper, our work complements these previous efforts and provides a richer understanding of how pre-trained mlms leverage context without assumptions on architectures. we aim to answer following questions: (i) how much context is relevant to and used by pre-trained mlms when composing representations? (ii) how far do mlms look when leveraging context? that is, what are their effective context window sizes? we further define a target word's essential context as the set of context words whose absence will make the mlm indiscriminate of its prediction. we analyze linguistic characteristics of these essential context words to better understand how mlms manage context. we investigate the contextual impacts in mlms via two approaches. first, we propose the context perturbation analysis methodology that gradually masks out context words following a predetermined procedure and measures the change in the target word probability. for example, we iteratively mask words that have the least change to the target word probability until the probability deviates too much from the start. at this point, the remaining words are relevant to and used by the mlm to represent the target word, since further perturbation causes a notable prediction change. being model agnostic, our approach looks into the contextualization in the mlm task itself, and quantify them only on the output layer. we refrain from inspecting internal representations since new architectures might not have a clear notion of "layer" with inter-leaving jump connections such as those in @xcite and @xcite . the second approach is adapted from @xcite and estimates the impact of an input subword to the target word probability via the norm of the gradients. we study pre-trained mlms based on two different architectures: transformer and bilstm. the former is essentially bert and the latter resembles elmo @xcite . although the scope in this work is limited to the comparison between two popular architectures, the same novel methodology can be readily applied to multilingual models as well as other transformerbased models pre-trained with mlm. from our analysis, when encoding words using sentence-level inputs, we find that bert is able to leverage 75% of context on average in terms of the sentence length, while bilstm has the effective context size of around 30%. the gap is compelling for long-range context more than 20 words away, wherein, bert still has a 65% chance to leverage the words in comparison to bilstm that only has 10% or less to do so. in addition, when restricted to a local context window around the target word, we find that the effective context window size of bert is around 78% of the sentence length, whereas bilstm has a much smaller window size of around 50%. with our extensive study on how different pre-trained mlms operate when producing contextualized representations and what detailed linguistic behaviors can be observed, we exploited these insights to devise a pilot application. we apply attention pruning that restricts the attention window of bert based on our findings. results show that the performance remains the same with its efficiency improved. our main contributions can be briefly summarized as: ## related work pre-training language models (lm) to learn contextualized word representations from a large amount of unlabeled text has been shown to benefit downstream tasks @xcite @xcite . masked language modeling (mlm) introduced in bert @xcite has been widely used as the pre-training task in works including roberta @xcite , spanbert @xcite , and albert @xcite . many of them employ the transformer architecture @xcite that uses multi-head self-attention to capture context. to assess the linguistic knowledge learned by pre-trained lms, probing task methodology suggest training supervised models on top of the word representations @xcite @xcite . investigated linguistic aspects span across morphology @xcite @xcite , syntax @xcite , and semantics @xcite . another line of research inspects internal states of pre-trained lms such as attention weights @xcite or intermediate word representations @xcite to facilitate our understanding of how pre-trained lms work. in particular, @xcite studies the evolution of representations from the bottom to top layers and finds that, for mlm, the token identity tends to be recreated at the top layer. a close work to us is @xcite , they conduct context analysis on lstm language models to learn how much context is used and how nearby and long-range context is represented differently. our work complements prior efforts by analyzing how models pre-trained by mlm make use of context and provides insights that different architectures can have different patterns to capture context. distinct from previous works, we leverage no specific model architecture nor intermediate representations while performing the context analysis. another related topic is generic model interpretations including lime @xcite , shap (lundberg and lee, 2017), and @xcite . despite the procedural similarity, our work focuses on analyzing how pre-trained mlms behave when encoding contexts and our methodology is both model-agnostic and training-free. @xcite . the comparable performance to previous works validates our pre-training process. ## masked language modeling given a sentence @xmath0, w 2 , ..., w l ) where each word w i is tokenized into l i subwords (s i1 , ..., s il i ), a portion of tokens are randomly masked with the [mask] token. mlms are trained to recover the original identity of masked tokens by minimizing the negative log likelihood (nll). in practice, bert @xcite randomly replaces 15% tokens by [mask] for 80% of the cases, keep the original token for 10% of the time, and replace with a random token for the remaining 10% of the cases. for context analysis, we perform the masking and predictions at the word level. given a target word w t , all its subwords are masked x \@xmath1...s (t-1)l t-1 , [mask], ..., [mask], s (t+1)1 ...). following @xcite , the conditional probability of w t can be computed from outputs of mlms with the independence assumption between subwords: to investigate how mlms use context, we propose procedures to perturb the input sentence from x \t to x \t and monitor the change in the target word probability p (w t |x \t ). ## approach our goal is to analyze the behaviors of pre-trained mlms when leveraging context to recover identity of the masked target word w t , e.g. to answer questions such as how many context words are considered and how large the context window is. to this end, we apply two analysis approaches. the first one is based on the masking or perturbation of input context which is architecture agnostic. the second gradient-based approach requires back-propagation through networks. our first approach performs context perturbation analysis on pre-trained lms at inference time and measures the change in masked target word probabilities. to answer each question, we start from x \t and design a procedure ψ that iteratively processes the sentence from last perturbation our second approach estimates the impact of an input subword s ij to p (w t |x \t ) by using derivatives. specifically, we adapt the impact score proposed in @xcite to our questions. the score impact(s ij , w t ) can be computed with the gradients of the negative log likelihood (nll) with respect to the subword embedding: the l 2 -norm of the gradient is used as the impact measure and normalized over all the subwords in a sentence. in practice, we report the impact of a context word w i by adding up the scores from its subwords l i j impact(s ij , w t ). we investigate two different encoder architectures of pre-trained mlms. the first one is bert that employs 12 transformer encoder layers, 768 dimension, 3072 feed-forward hidden size, and 110 million parameters. the other uses a standard bidirectional lstm (hochreiter and schmidhuber, 1997) that has 3 layers, 768 embedding dimension, 1200 hidden size, and around 115 million parameters. the bilstm model parameters are chosen so that they resemble elmo while being close to bert in model size. to have a fair comparison, we pre-train both encoders from scratch on the uncased wikipedia-book corpus (wikibook) with the same pre-training setup as in @xcite . for bilstm, we add a linear layer and a layernorm @xcite on top, to project outputs into 768 dimension. we validate our pre-trained models by fine-tuning them on glue benchmark @xcite in single-task manner and report test performance comparable to previous works in our pre-trained bilstm-based mlm also gets comparable results to elmo @xcite . we perform mlm context analysis on two english datasets from the universal dependencies (ud) project, english web treebank (ewt) @xcite and georgetown university multilayer corpus (gum) @xcite . datasets from the ud project provide consistent and rich linguistic annotations across diverse genres, enabling us to gain insights towards the contexts in mlms. we use the training set of each dataset for analysis. ewt consists of 9, 673 sentences from web blogs, emails, reviews, and social media with the median length being 17 and maximum length being 159 words. gum comprises 3, 197 sentences from wikipedia, news articles, academic writing, fictions, and how-to guides with the median length being 19 and maximum length being 98 words. the statistics of datasets are summarized in self-attention is designed to encode information from any position in a sequence, whereas bil-stms model context through the combination of long-and short-term memories in both left-to-right and right-to-left directions. for mlms, the entire sequence is provided to produce contextualized representations, it is unclear how much context in the sequence is used by different mlms. in this section, we first propose a perturbation procedure ψ that iteratively masks out a context word contributing to the least absolute change of the target word probability p (w t | x k \t ). that is, we incrementally eliminate words that do not penalize mlms predictions one by one, until further masking cause p (w t | x k \t ) to deviate too much from the original probability p (w t |x \t ). at this point, the remaining unmasked words are considered being used by the mlm since corrupting any of them causes a notable change in target word prediction. in practice, we identify deviations using the negative log likelihood (nll) that corresponds to the loss of mlms. assuming nll has a variance of at the start of masking, we stop the perturbation procedure when the increase on nll log p (w t |x \t ) -log p (w t | x k \t ) exceeds 2 . we observe that nlls fluctuate around [-0.1, 0.1] at 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 the start of masking, hence we terminate our procedure when the nll increase reaches 0.2. we report the effective context size in terms of percentage of length to normalize the length impact. the analysis process is repeated using each word in a sentence as the target word for all sentences in the dataset. for our second approach, we follow equation 2 to calculate the normalized impact of each subword to the target word and aggregate them for each context word to get impact(w i , w t ). we group the im-pact scores by relative position of a word w i to the target word w t and plot the average. to compare with our first approach, we also use masking-based method to analyze that for a word with a specific relative position, what would be its probability of being used by a mlm. bert uses distant context more than bilstm. after our masking process, a subset of context words are tagged as "being used" by the pre-trained lm. in figure 1a , we aggregated results in terms of relative positions (context-word-to-target-word) for all targets and sentences. "probability of being used %" denotes when a context word appears at a relative position to target, how likely is it to be relevant to the pre-trained lm. figure 1a shows that context words at all relative positions have substantially higher probabilities to be considered by bert than bilstm. and bil-stm focuses sharply on local context words, while bert leverages words at almost all the positions. a notable observation is that both models consider a lot more often, words within distance around @xcite and bert has as high as 90% probability to use the words just before and after the target word. using gradient-based analysis, figure 1b shows similar results that bert considers more distant context than bilstm and local words have 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 more impact to both models than distant words. there are notable differences between two analysis approaches. since the gradient-based im-pact score is normalized into a distribution across all positions, it does not show the magnitude of the context impact on the two different models. on the other hand, the masking-based analysis shows that bert uses words at each position more than bilstm based on absolute probability values. another important difference is that the gradientbased approach is a glass-box method and requires back-propagation through networks, assuming the models to be differentiable. on the other hand, the masking-based approach treats the model as a black-box and has no differentiability assumption on models. in the following sections, we will continue analysis with the masking-based approach. bert uses 75% of words in a sentence as context while bilstm considers 30%. figure 2 shows the increase in nll when gradually masking out the least relevant words. bert's nll increases considerably when 25% of context are masked, suggesting that bert uses around 75% of context. for bilstm, its nll goes up remarkably after 70% of context words are masked, meaning that it considers around 30% of context. albeit having the same capacity, we observe that bert uses more than two times of context words into account than bilstm. this could explain the superior fine-tuning performance of bert on tasks demanding more context to solve. we observe that pre-trained mlms have consistent behaviors across two datasets that have different genres. for the following analysis, we report results combining ewt and gum datasets. content words needs more context than function words. we bucket instances based on the part-of-speech (pos) annotation of the target word. our analysis covers content words including nouns, verbs and adjectives, and function words including adpositions and determiners. figure 3a shows that both models use significantly more context to represent content words than function words, which is aligned with linguistic intuitions @xcite . the findings also show that mlms handle content and function words in a similar manner as regular language models do, which are previously analyzed by @xcite ; @xcite . bilstm context usage percentage varies by input sentence length, whereas for bert, it doesn't. we categorize sentences with length shorter than 25 as short, between 25 and 50 as medium, and more than 50 as long. figure 3b shows that bilstm uses 35% of context for short sentences, 20% for medium, and only 10% for long sentences. on the other hand, bert leverages fixed 75% of context words regardless of the sentence length. in the previous section, we looked at how much context is relevant to the two mlms via an elimination procedure. from figure 1a and 1b, we also observe that local context is more impactful than long-range context for mlms. in this section, we investigate this notion of locality of context even further and try to answer the question of how far away do mlms actually look at in practice, i.e., what is the effective context window size (cws) of each mlm. for context perturbation analysis, we introduce a locality constraint to the perturbation procedure 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 access to combined context window size on both sides (% of available context) while masking words. we aim to identify how local versus distant context impacts the target word probability differently. we start with masking all the words around the target, i.e., the model only relies on its priors learned during pre-training (cws ∼ 0% foot_0 ). we iteratively increase the cws on both sides until all the surrounding context is available (cws ∼ 100%). details of the masking procedure can be found in appendix. we report the increase in nll compared to when the entire context is available log p (w t |x \t ) -log p (w t | x k \t ), with respect to the increasing cws. this process is repeated using each word as the target word, for all the sentences in the dataset. we aggregate and visualize the results similar to section 5 and use the same threshold (0.2) as before to mark the turning point. as shown in figure 4 , increasing the cws around target word reduces the change of nll until a point where the gap is closed. the plot clearly highlights the differences in the behavior of two modelsfor bert, words within cws of 78% impact the model's ability to make target word predictions, whereas, for bilstm, only words within cws of 50% affect the target word probability. this shows that bert, leveraging entire sequence by selfattention, looks at a much wider context window size (effective cws ∼ 78%) in comparison to the recurrent architecture bilstm (effective cws ∼ 50%). besides, bilstm shows a clear notion of contextual locality that it tends to consider very local context for target word prediction. furthermore, we investigate the symmetricity of cws on either side by following the same procedure but now separately on each side of the target word. we iteratively increase cws either on left side or right side while keeping the rest of the words unmasked. more details of the analysis procedure can be found in the appendix. the analysis results are further bucketed by the pos categories of target words as well as input sentence lengths, similar to section 5, to gain more fine-grained insights. in figure 5 , we show the symmetricity analysis of cws for short length sentences and target word with pos tags -noun and det. the remaining plots for medium and long length sentences with target word from other pos tags are shown in appendix due to the lack of space. from figure 5 , both models show similar behaviors across different pos tags when leveraging symmetric/asymmetric context. the cws attended to on either side is rather similar when target words are noun, whereas for det, we observe both models paying more attention to right context words than the left. this observation aligns well with linguistic intuitions for english language. we can also observe the striking difference between two models in effective cws, with bert attending to a much larger cws than bilstm. the difference in 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 the left and right cws for det appears to be more pronounced for bilstm in comparison to bert. we hypothesize that this is due to bilstm's overall smaller cws (left + right) which makes it only attend to the most important words that happen to be mostly in the right context. ## what kind of context is essential? there is often a core set of context words that is essential to capture the meaning of target word. for example, "many people think cotton is the most comfortable to wear in hot weather." although most context is helpful to understand the masked word fabric, cotton and wear are essential as it would be almost impossible to make a guess without them. in this section, we define essential context as words such that when they are absent, mlms would have no clue about the target word identity, i.e., the target word probability becomes close to masking out the entire sequence p (w t | x mask all ). to identify essential context, we design the perturbation ψ to iteratively mask words bringing largest drop in p (w t | x k \t ) until we reach a point, where the increase in nll just exceeds the 100% mask setting (log the words masked using above procedure are labelled as essential context words. we further analyze linguistic characteristics of the identified essential context words. bert sees 35% of context as essential, whereas bilstm perceives around 20%. figure 6 shows that on average, bert recognizes around 35% of context as essential when making predictions, i.e., when the increase in nll is on par with masking all context. on the other hand, bilstm sees only 20% of context as essential. this implies that bert would be more robust than the bilstmbased encoder in the presence of noisy input, a finding also supported by @xcite ; @xcite , as it will be harder to confuse the model completely given larger size of essential context words set in comparison to bilstm. essential words are close to target words in both linear position and on dependency tree. figure 7 shows examples of essential words from bert with pos tags and dependency relations. words in square brackets are target words and the underlined words are essential words. we observe that words close to the target in the sentence as well as in the dependency tree are often seen as essential. we can also see that bert often includes the root of the dependency tree as an essential word. ## application: attention pruning for transformer as a pilot application, we leverage insights from analysis in previous sections to perform attention pruning for transformer. transformer has achieved impressive results in nlp and has been used for long sequences with more than 10 thousand tokens @xcite . self-attention for a sequence of length l is of o(l foot_1 ) complexity in computation and memory. many works attempt to improve the efficiency of self-attention by restricting the number of tokens that each input query can attend to @xcite . our analysis in section 6 shows that bert has effective cws of around 78%. we perform a dynamic attention pruning by making self-attention neglect the furthest 22% of tokens. due to the o(l 2 ) complexity, this could save around 39% of computation in self-attention. we apply this locality constraint to self-attention when fine-tuning bert on a downstream task. specifically, we use the conll-2003 named entity recognition (ner) dataset @xcite with 200k words for training. we fine-tune bert for ner in the same way as in @xcite . we also explore a static attention pruning that restricts the attention span to be within [-5, +5] 2 . results in ## conclusion in our context analysis, we have shown that bert has an effective context size of around 75% of input length, while bilstm has about 30%. the difference in context usage is striking for long-range context beyond 20 words. our extensive analysis of context window size demonstrate that bert uses much larger context window size than bilstm. besides, both models often identify words with common syntactic structures as essential context. these findings not only help to better understand contextual impact in masked language models, but also encourage model improvements in efficiency and effectiveness in future works. on top of that, diving deep into the connection between our con-text analysis and a model's robustness to noisy texts is also an interesting topic to explore.