Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2021)

Abstract: A growing body of work makes use of probing in order to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer behavioral conclusions from probing results, and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be assessed by measuring the influence of a causal intervention that removes it from the representation. Equipped with this new analysis tool, we can ask questions that were not possible before, for example, is part-of-speech information important for word prediction? We perform a series of analyses on BERT to answer these types of questions. Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.1

## introduction what drives a model to perform a specific prediction? what information is being used for prediction, and what would have happen if that information went missing? because neural representation is opaque and hard to interpret, answering these questions is challenging. the recent advancements in language models (lms) and their success in transfer learning of many nlp tasks (e.g., @xcite @xcite spiked interest in understanding how these models work and what is being encoded in them. one prominent methodology that attempts to shed light on those questions is probing @xcite (also known as auxilliary prediction @xcite and diagnostic classification @xcite ). under this methodology, one trains a simple model -a probe-to predict some desired information from the latent representations of the pre-trained model. high prediction performance is interpreted as evidence for the information being encoded in the representation. a key drawback of such an approach is that while it may indicate that the information can be extracted from the representation, it provides no evidence for or against the actual use of this information by the model. indeed, @xcite have shown that under certain conditions, above-random probing accuracy can be achieved even when the information that one probes for is linguistically meaningless noise, which is unlikely to have any use by the actual model. more recently, @xcite showed that models encode linguistic properties, even when not required at all for solving the task, questioning the usefulness and common interpretation of probing. these results call for higher scrutiny of causal claims based on probing results. in this paper, we propose a counterfactual approach that serves as a step towards causal attribution: amnesic probing (see figure 1 for a schematic view). we build on the intuition that if a property z (e.g., part-of-speech) is being used for a task t (e.g., language modeling), then the removal of z should negatively influence the ability of the model to solve the task. conversely, when the removal of z has little or no influence on the ability to solve t , one can argue that knowing z is not a significant contributing factor in the strategy the model employs in solving t . as opposed to previous work that focused on intervention in the input space @xcite figure 1 : a schematic description of the proposed amnesic intervention: we transform the contextualized representation of the word ''ran'' so as to remove information (here, pos), resulting in a ''cleaned'' version h ¬p os ran . this representation is fed to the wordprediction layer and the behavioral influence of pos erasure is measured. @xcite or in specific neurons @xcite , our intervention is done on the representation layers. this makes it easier than changing the input (which is non-trivial) and more efficient than querying hundred of neurons (which become combinatorial when considering the effect of multiple neurons simultaneously). we demonstrate that amnesic probing can function as a debugging and analysis tool for neural models. specifically, by using amnesic probing we show how to deduce whether a property is used by a given model in prediction. in order to build the counterfactual representations, we need a function that operates on a pretrained representation and returns a counterfactual version which no longer encodes the property we focus on. we use the recently proposed algorithm for neutralizing linear information: iterative nullspace projection (inlp) @xcite . this approach allows us to ask the counterfactual question: ''how will the prediction of a task differ without access to some property?'' @xcite . this approach relies on the assumption that the usefulness of some information can be measured by neutralizing it from the representation, and witnessing the resulting behavioral change. it echoes the basic idea of ablation tests where one removes some component and measures the influence of that intervention. we study several linguistic properties such as part-of-speech (pos) and dependency labels. overall, we find that as opposed to the common belief, high probing performance does not mean that the probed information is used for predicting the main task ( §4). this is consistent with the recent findings of @xcite . our analysis also reveals that the properties we examine are often being used differently in the masked setting (which is mostly used in lm training) and in the non-masked setting (which is commonly used for probing or fine-tuning) ( §5). we then dive deeper into a more fine-grained analysis, and show that not all of the linguistic property labels equally influence prediction ( §6). finally, we reevaluate previous claims about the way that bert process the traditional nlp pipeline @xcite with amnesic probing and provide a novel interpretation on the utility of different layers ( §7). ## to probe or not to probe? by using the probing technique, different linguistic phenomenon such as pos, dependency information, and ner @xcite @xcite have been found to be ''easily extractable'' (typically using linear probes). a naive interpretation of these results may conclude that because information can be easily extracted by the probing model, this information is being used for the predictions. we show that this is not the case. some properties such as syntactic structure and pos are very informative and are being used in practice to predict words. however, we also find some properties, such as phrase markers, which the model does not make use of when predicting tokens, in contrast to what one can naively deduce from probing results. this finding is in line with a recent work that observed the same behavior @xcite . for each linguistic property, we report the probing accuracy using a linear model, as well as the word prediction accuracy after removing information about that property. the results are summarized in these results strengthen recent works that question the usefulness of probing as an analysis tool @xcite we report the vanilla word prediction accuracy and the amnesic scores, as well as the rand and 1-hot controls which shows minimal information loss and high selectivity (except for the dep property which all information was removed). the d kl is also reported for all properties in the last rows which show similar trends as the accuracy performance. 2020 ), but measure it from the usefulness of properties on the main task. we conclude that high probing performance does not entail this information is being used at a later part of the network. probing studies tend to focus on representations that are used for an end-task (usually the last hidden layer before the classification layer). in the case of mlm models, the words are not masked when encoding them for downstream tasks. however, these representations are different from those used during the pre-training lm phase (of interest to us), where the input words are masked. it is therefore unclear if the conclusions drawn from conventional probing also apply to the way that the pre-trained model operates. from this section on, unless mentioned otherwise, we report our experiments on the masked words. that is, given a sequence of tokens x 1 , . . . , x i , . . . , x n we encode the representation of each token x i using its context, as follows: x 1 , . . . , x i-1 , [mask], x i+1 , . . . , x n . the rest of the tokens remain intact. we feed these input tokens to bert, and only use the masked representation of each word in its context h(x 1 , . . . , x i-1 , [mask], x i+1 , . . . , x n ) i . we repeat the experiments from section 4 and report the results in trends in the masked setting are similar to the nonmasked setting. however, this is not always the case, as we show in section 7. we also report the selectivity control. notice that the performance for this experiment was improved across all tasks. in the case of dep and f-pos, where we had to neutralize most of the dimensions the performance does not fully recover. note that the number of classes in those experiments might be a factor in the large performance gaps (expressed by the number of removed dimensions, n. dir, in the table). while not part of this study, it would be interesting to control for this factor in future work. however for the rest of the properties (c-pos, ner, and the phrase-markers) the performance is fully recovered, showing our methods' selectivity. to further study the effect of inlp and inspect how the different dimensions removal affect performance, we display in figure 2 the lm performance after each iteration, both with the amnesic probing and the control, and observe a consistent gap between them. moreover, we highlight the difference in the slope for our method and the random direction removal. the amnesic probing exemplifies a much steeper slope than the random direction, indicating that the studied properties are indeed correlated with words prediction. we also provide the main task performance after each iteration in figure 5 in the appendix, which steadily decreases with each iteration. ## specific labels and word prediction in the previous sections we observed the impact (or lack thereof) of different properties on word prediction. but when a property affects words prediction, are all words affected similarly? in this section, we inspect a more fine-grained version of the properties of interest, and study the impact of those on word predictions. ## behavior across layers the results up to this section treat all of bert's 'transformer blocks' @xcite as the encoding function and the embedding matrix as the model. but what happens when we remove the information of some linguistic property from earlier layers? by using inlp to remove a property from an intermediate layer, we prevent the subsequent layer from using linearly present information originally stored in that layer. though this operation does not erase all the information correlative with the studied property (as inlp only removes linearly present information), it makes it harder for the model to use this information. concretely, we begin by extracting the representation of some text from the first k layers of bert and then run inlp on these representations to remove the property of interest. given that we wish to study the effect of a property on layer i, we project the representation using the corresponding projection matrix p i that was learned on those representation, and then continue the encoding of the following layers. 12 ## related work with the established impressive performance of large pre-trained language models @xcite , based on the transformer architecture @xcite , a large body of work started studying and gaining insight into how these models work and what do they encode. 15 for a thorough summary of these advancements we refer the reader to a recent primer on the subject @xcite . a particularly popular and easy-to-use interpretation method is probing @xcite . despite its popularity, recent works have questioned the use of probing as an interpretation tool. @xcite have emphasized the need to distinguish between decoding and learning the probing tasks. they introduced control tasks, a consistent but linguistically meaningless attribution of labels to tokens, and have shown that probes trained on the control tasks often perform well, due to the strong lexical information held in the representations and learned by the probe. this leads them to propose a selectivity measure that aims to choose probes which achieve high accuracy only on linguistically-meaningful tasks. tamkin et al. (2020) claim that probing cannot serve as an explanation of downstream task success. they observe that the probing scores do not correlate with the transfer scores achieved by fine-tuning. finally, @xcite show that probing can achieve non-trivial results for linguistic properties that were not needed for the task the model was trained on. in this work, we observe a similar phenomenon, but from a different angle. we actively remove some property of interest from the queried representation, and measure the impact of the amnesic representation of the property on the main task. two recent works study the probing paradigm from an information-theory perspective. @xcite emphasize that under a mutualinformation maximization objective, ''better'' probes are increasingly more accurate, regardless of complexity. they use the data-processing inequality to question the rationale behind methods that focus on encoding, and propose ease of extractability as an alternative criterion. @xcite follow this direction, using the concept of minimum description length (mdl, @xcite to quantify the total information needed to transmit both the probing model and the labels it predicts. our discussion here is somewhat orthogonal to those on the meaning of encoding and probe complexity, as we focus on the information influence on the model's behavior, rather than on the ability to extract it from the representation. finally and concurrent to this work, feder et al. (2020) have studied a similar question of a causal attribution of concepts to representations, using adversarial training guided by causal graphs. ## discussion intuitively, we would like to completely neutralize the abstract property we are interested in-e.g., pos information (completeness), as represented by the model-while keeping the rest of the representation intact (selectivity). this is a nontrivial goal, as it is not clear whether neural models actually have abstract and disentangled representations of properties such as pos, which are independent of other properties of the text. it may be the case that the representation of many properties is intertwined. indeed, there is an ongoing debate on the assertion that certain information is ''encoded'' in the representation @xcite . however, even if a disentangled representation of the information we focus on exists, it is not clear how to detect it. we implement the information removal operation with inlp, which gives a first order approximation using linear classifiers; we note, however, that one can in principle use other approaches to achieve the same goal. while we show that we do remove the linear ability to predict the properties and provide some evidence to the selectivity of this method ( §2), one has to bear in mind that we remove only linearly-present information, and that the classifiers can rely on arbitrary features that happen to correlate with the gold label, be it a result of spurious correlations or inherent encoding of the direct property. indeed, we observe this behavior in section 7.1 (figure 3 ), where we neutralize the information from certain layers, but occasionally observe higher probing accuracy in following layers. we thus stress that the information we remove in practice should be seen only as an approximation for the abstract information we are interested in, and that one has to be cautious of causal interpretations of the results. although in this paper we use the inlp algorithm in order to remove linear information, amnesic probing is not restricted to removing linear information. when non-linear removal methods become available, they can be swapped instead of inlp. this stresses the importance of creating algorithms for non-linear information removal. another unanswered question is how to quantify the relative importance of different properties encoded in the representation for the word prediction task. the different erasure portion for different properties makes it hard to draw conclusions on which property is more important for the task of interest. although we do not make claims such as ''dependency information is more important than pos'', these are interesting questions that should be further discussed and researched. ## conclusions in this work, we propose a new method, amnesic probing, which aims to quantify the influence of specific properties on a model that is trained on a task of interest. we demonstrate that conventional probing falls short in answering such behavioral questions, and perform a series of experiments on different linguistic phenomenon, quantifying their influence on the masked language modeling task. furthermore, we inspect both unmasked and masked bert's representation and detail the differences between them, which we find to be substantial. we also highlight the different influence of specific fine-grained properties (e.g., nouns and determiners) on the final task. finally, we use our proposed method on the different layers of bert, and study which parts of the model make use of the different properties. taken together, we argue that compared with probing, counterfactual intervention-such as the one we present here-can provide a richer and more refined view of the way symbolic linguistic information is encoded and used by neural models with distributed representations. 16