How does BERT ’s attention change when you fine-tune? An analysis methodology and a case study in negation scope (2020)

Abstract: Large pretrained language models like BERT, after fine-tuning to a downstream task, have achieved high performance on a variety of NLP problems. Yet explaining their decisions is difficult despite recent work probing their internal representations. We propose a procedure and analysis methods that take a hypothesis of how a transformer-based model might encode a linguistic phenomenon, and test the validity of that hypothesis based on a comparison between knowledge-related downstream tasks with downstream control tasks, and measurement of cross-dataset consistency. We apply this methodology to test BERT and RoBERTa on a hypothesis that some attention heads will consistently attend from a word in negation scope to the negation cue. We find that after fine-tuning BERT and RoBERTa on a negation scope task, the average attention head improves its sensitivity to negation and its attention consistency across negation datasets compared to the pre-trained models. However, only the base models (not the large models) improve compared to a control task, indicating there is evidence for a shallow encoding of negation only in the base models.

## introduction as large-scale pre-trained language models such as bert and elmo have achieved high performance in a variety of natural language processing tasks @xcite @xcite , a growing body of research is devoted to understanding what linguistic properties these language models have acquired. recent work uses probes, which are supervised models trained to predict linguistic properties including morphology @xcite , syntax @xcite and semantics @xcite , etc. (see @xcite for a complete survey.) a good probing performance is considered as evidence that the language models have learned the linguistic knowledge. what is not yet well understood is how this encoded linguistic knowledge changes when a pretrained language model is fine-tuned for a downstream task. @xcite applies a supervised probe both before and after fine-tuning bert, and suggests that fine-tuning makes the internal representation task-sensitive. but with supervised probes it can be difficult to disentangle what was learned by the probe from what was present in the internal representation @xcite . recent studies have thus turned to unsupervised probes that require no additional training of the model and instead look directly at the attention mechanism, i.e., how much to care about other words when computing the next version of the current word. @xcite inspected pretrained transformers and found several syntactic properties encoded in an intuitive way, where the maximum attention from a dependent is on its syntactic head. but only the pretrained models were considered, not what happened to these intuitive encodings after fine-tuning to a downstream task. we argue that if some interpretable encoding of linguistic knowledge is a good explanation of a model, rather than showing it in the pretrained model, it is more important to show it will be enhanced by fine-tuning on a task where that linguistic knowledge is necessary. if the encoding is not enhanced by such fine-tuning, then the model must be using some other mechanism to encode that linguistic knowledge. we therefore propose the following methodology for testing whether a hypothesized encoding of a linguistic phenomenon is a good explanation for a transformer's predictions. applying this methodology and a variety of analyses that it enables, and focusing on the phenomenon of linguistic negation scope in a intuitive encoding (the maximal attention from a word in negation scope will be on the negation cue), we find that: items 1 and 2 suggest that in the pretrained models negation scope may be encoded via attention to negation cues. items 3 to 5 indicate that during fine-tuning, this encoding continues to play a role in bert-base and roberta-base, but robertalarge and bert-large may rely on other mechanisms to represent negation scope. the analysis code is available at https://github.com/ yiyunzhao/negation-scope-probing though our findings are specific to the linguistic phenomenon of negation scope and the specific attention encoding we hypothesized, our proposed methodology and analyses are general, and can easily be applied to other linguistic phenomena or other encoding hypotheses to discover the role they play in modern pre-trained neural network models. ## methodology and analyses in this section, we explain our proposed methodology and analyses, and illustrate their application to the linguistic phenomenon of negation scope. step 1: hypothesize an interpretable representation of the phenomenon of interest. transformer models could represent linguistic knowledge in many ways: attention, contextualized embeddings, etc. to apply our methodology, one must first hypothesize a specific encoding of the phenomenon of interest. for negation scope, we hypothesize that for some subset of attention heads, words in negation scope will attend primarily to the negation cue, while words out of negation scope will attend primarily to other words (see section 4.1). under this hypothesis, each attention head is an unsupervised negation scope classifier. ## discussion we have presented a methodology for looking for explanations of transformer models, where a hypothesized encoding of knowledge within the transformer is measured before and after fine-tuning and the changes are compared to those seen when finetuning on a control task. we considered a specific linguistic phenomenon, negation scope detection, proposed an intuitive way that attention may encode negation-scope (in-scope words pay attention to the negation cue), and applied our methodology to test whether the hypothesized encoding was indeed an explanation of the behavior of bert and/or roberta models. we found evidence that bert-base and roberta-base encode some negation knowledge in the proposed way as both average negation sensitivity and cross-dataset consistency improved over the pretrained model and the control task. evidence for the large versions of the models was weaker, suggesting that they may be representing negation knowledge in other ways. other works have explored the effects of finetuning on attention without testing for specific linguistic knowledge. @xcite , @xcite and @xcite found many redundancies in the attention of sequence-to-sequence models, suggesting that attention may encode knowledge in many ways. @xcite found that removal of attention heads in transformers does not necessarily damage downstream performance. our results suggest an explanation for this finding: knowledge sensitivity spreads broadly, so recovering from a small number of missing heads should be easy. other work has tested for specific linguistic knowledge in pretrained models, but not explored how the encoding of that knowledge changes during fine-tuning. for instance, @xcite identified several syntactic relationships that are encoded in an intuitive way: the dependent's primary attention is on its grammtical head. we argue that testing whether this hypothesized encoding of grammatical relations survives fine-tuning is critical if this is to be an explanation of how transformer models make predictions. we found no past work that considered the crossdataset consistency of attention. we believe measuring such consistency is important for differentiating between an attention head that learned to encode a linguistic phenomenon for a single dataset vs. an attention head that learned an encoding of the true linguistic phenomenon. for example, it could have been the case that fine-tuning improves sensitivity to negation in both datasets, but the improvements happen at different heads. we see this for example in bert-large on the control task, where there is essentially zero consistency in which atten-tion heads are active across the two datasets. some limitations of our current work suggest future research directions. first, we have focused on one interpretable way of encoding of negation scope knowledge but one can hypothesize many other ways. for instance, instead of assuming that all in-scope words directly pay attention to negation cue, it is possible that the head of in-token words are organized in a tree of attention that leads to the negation cue. we use a single nonlinguistic control task, but one could imagine exploring attention head changes in the face of a gradient of fine-tuning tasks that are more or less relevant to the linguistic phenomenon of interest. we also focus primarily on the attention mechanism, but it would be useful to explore the value vectors that transformers apply the attention to, since these form the outputs and are thus more directly tied to classification decisions. ## conclusion in this paper, we propose a basic procedure and analysis methods that take a hypothesis of how a transformer-based model might encode a linguistic phenomenon, and test the validity of that hypothesis based on unsupervised probes, downstream control tasks, and measurement of cross-dataset consistency. we hypothesize an interpretable encoding of negation scope, where in-scope words attend to the negation cue, and find evidence of such an encoding in bert-base and roberta-base. figure a4 : the heatmap of unsupervised negation-scope classification performance for roberta-large's 24 layers x 16 heads across two different datasets. the consistency (measure by kendall rank correlation) between the two datasets for precision, recall and f 1 are 0.377, 0.504 and 0.493 respectively.