Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021)

Abstract: Although neural models have achieved impressive results on several NLP benchmarks, little is understood about the mechanisms they use to perform language tasks. Thus, much recent attention has been devoted to analyzing the sentence representations learned by neural encoders, through the lens of ‘probing’ tasks. However, to what extent was the information encoded in sentence representations, as discovered through a probe, actually used by the model to perform its task? In this work, we examine this probing paradigm through a case study in Natural Language Inference, showing that models can learn to encode linguistic properties even if they are not needed for the task on which the model was trained. We further identify that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself, highlighting the importance of careful controls when designing probing experiments. Finally, through a set of controlled synthetic tasks, we demonstrate models can encode these properties considerably above chance-level, even when distributed in the data as random noise, calling into question the interpretation of absolute claims on probing tasks.

## introduction neural models have established state-of-the-art performance on several nlp benchmarks @xcite @xcite . however, these models can be opaque and difficult to interpret, posing barriers to widespread adoption and deployment in safety-critical or userfacing settings @xcite . how can we know what information, if any, neural models learn and leverage to perform a task? this ques- sentence representation auxiliary task 2. freeze 3. probe model control dataset sentence representation 1. train 2. freeze probe acc. auxiliary task 3. probe outcome: similar probing accuracy for model trained on main dataset or control dataset. model trained on main dataset model trained on control dataset ## background and related work progress in natural language understanding (nlu) has been driven by a history of defining tasks and corresponding benchmarks for the community @xcite @xcite . these tasks are often tied to specific practical applications, or to developing models demonstrating competencies that transfer across applications. the corresponding benchmark datasets are utilized as proxies for the tasks themselves. how can we estimate their quality as proxies? while annotation artifacts are one facet that affects proxy-quality @xcite @xcite @xcite , a dataset might simply not have coverage across competencies required for a task. additionally, it might consist of alternate "explanations", features correlated with the task label in the dataset while not being taskrelevant, which models can exploit to give the impression of good performance at the task itself. two analysis methods have emerged to address this limitation: 1) diagnostic examples, where a small number of samples in a test set are annotated with linguistic phenomena of interest, and task accuracy is reported on these samples @xcite . however, it is difficult to determine if models perform well on diagnostic examples because they actually learn the linguistic competency, or if they exploit spurious correlations in the data @xcite @xcite . 2) external challenge tests @xcite @xcite @xcite , where examples are constructed, either through automatic methods or by experts, exercising a specific phenomenon in isolation. however, it is challenging and expensive to build these evaluations, and non-trivial to isolate phenomena @xcite . thus, probing or diagnostic classification presents a compelling alternative, wherein learned representations can directly be probed for linguistic properties of interest @xcite @xcite @xcite . there has been a variety of research that employs probing to test hypotheses about the mechanisms models used to perform tasks. @xcite examine learned representations in machine translation for syntactic knowledge. @xcite investigate aspect in neural machine translation systems, finding that tense information could be extracted from the encoder, but that part of this information may be lost when decoding. @xcite use probing to examine the correlation between linguistic properties and downstream tasks (including mt and nli). hupkes et al. (2018) train a 'diagnostic classifier' to extract information from a sequence of hidden representations in a neural network. if the classifier achieves high accuracy, it is concluded that the network is keeping track of the hypothesized information. giulianelli et al. (2018) use diagnostic classifiers to predict number from the internal states of a language model. kim et al. (2019) study what different nlp tasks teach models about function word comprehension. alt et al. (2020) analyze learned representations in relation extraction, through a set of fourteen probing tasks for relevant linguistic properties. saleh et al. (2020) examine the representations learned by neural dialog models for insights into what the model learns about engaging in dialog. see the survey by belinkov and glass (2019) for many more examples. closely related to our work is that of @xcite , which studies the role of lexical memorization in probing, and recently the work of @xcite and @xcite who analyze probing from an information-theoretic perspective. these works join an ongoing debate on the correct way to characterize the expressivity of the probing classifier, with the latter proposing ease of extractability as a criterion for selecting appropriate probes. our work pursues an orthogonal line of inquiry, demonstrating that relying on diagnostic classifiers to interpret model reasoning for a task suffers from a fundamental limitation: properties may be incidentally encoded even when not required for a task. thus, our work is also related to a broader investigation of how neural models encode information @xcite , studying to what extent information encoded in neural representations is indicative of information needed to perform tasks. ## methodology in this section we describe our modified probing pipeline (figure 1 ), where we construct control datasets, such that a particular linguistic feature is not required in making task judgements. 2 control datasets are based on the intuition that a linguistic feature is not informative for a model to discriminate between classes if the linguistic feature remains constant across classes. for a task label t and linguistic property l, when every example in the control dataset has the same value for l, the linguistic property l in isolation is not discriminative of the task label . to construct control datasets we hold constant the relevant property value across the whole dataset. in practice, the control datasets are constructed from existing large-scale datasets by partitioning them on the value of a linguistic property, such that every example in the sampled dataset has the same value of linguistic property. 3 they are designed with the following considerations: main task: we study the natural language inference (nli) training task from @xcite as the main task for training sentence encoders. nli is a benchmark task for research on natural language understanding @xcite @xcite @xcite @xcite @xcite . broadly, the goal of the task is to decide if a given hypothesis can be inferred from a premise in a justifiable manner. typically, this is framed as the 3-way decision of whether a hypothesis is true given the premise (entailment), false given the premise (contradiction), or whether the truth value cannot be determined (neutral). we use multinli @xcite , a broad-coverage nli dataset, to train sentence encoders. auxiliary tasks: we consider three tasks that probe sentence representations for semantic information from @xcite , all of which "require some understanding of what the sentence denotes". we construct the probing datasets such that lexical items that are associated with the probing task do not occur across the train/dev/test split for the target. this design controls for the effect of memorizing word types associated with target categories @xcite . the tasks considered in this study are: control: for each auxiliary task, we partition multinli such that premises and hypotheses agree on a single value of the linguistic property. for example, for the auxiliary task tense, sentences with vbp/vbz/vbg forms are labeled as present and vbd/vbn as past tense. 4 subsequently, premisehypothesis pairs where the main verbs in both premise and hypothesis are in past tense are extracted from train/dev sets to form the control datasets for tense. thus, every sentence in the dataset (both premises and hypotheses), has the same value of the auxiliary property. 5 this procedure results in three control datasets/tasks: multinli-pasttense, multinli-singularsubject, and multinli-singularobject. for all three, we fix the value of the linguistic property to the one that results in the maximum number of training instances on partitioning, namely fixing past tense, singular subject number, and singular object number. descriptive statistics for each dataset appears in models: we use cbow and bilstm-based sentence-encoder architectures. the choice of these models is motivated by their demonstrated utility as nli architectures @xcite , and because their learned representations have been extensively studied for the three linguistic properties used in this work @xcite . 6 1. majority: the hypothetical performance of a classifier that always predicts the most frequent label in the test set. all models produce separate sentence vectors for the premise and hypothesis. they are concatenated with their element-wise product and difference @xcite , passed to a tanh layer and then to a 3-way softmax classifier. models are initialized with 300d glove embeddings @xcite unless specified otherwise, and implemented in dynet @xcite . after the model is trained for the nli task, the learned sentence vectors for the premise and hypothesis are probed. the probing classifier is a 1-layer multilayered perceptron (mlp) with 200 hidden units. ## a synthetic experiment: analyzing encoding dynamics we have demonstrated that models encode properties even when they are not required for the main task. thus, probing accuracy cannot be considered indicative of competencies any given model relies on. what circumstances could lead to models encoding properties incidentally? can we determine when a linguistic property is not needed by a model for a task? to study this, we build carefully controlled synthetic tests, each capturing a kind of noise that could arise in datasets. ## discussion we briefly discuss our findings, with the goal of providing considerations for deciding which inferences can be drawn from a probing study, and highlighting avenues for future research. linguistic properties can be incidentally encoded: probing only indicates that some property correlated with a linguistic property of interest is encoded in the sentence representation -but we speculate that it cannot isolate what that property might be, whether the correlation is meaningful, or how many such properties exist. as shown in the controlled synthetic tests, even if a particular property is not needed for a task, the information can be extracted from the representation with high accuracy. thus, probing cannot determine if the property is actually needed to do a task, and should not be used to pinpoint the information a model is relying upon. a negative result here can be more meaningful than a positive one. adversarially suppressing the property may help determine if an alternate explanation is readily available to the model, with an appropriate choice of probing classifier. in this case, if the model maintains task accuracy while suppressing the information, one can conclude the property is not needed by the model for the task, but its failure to do so is not indicative of property importance. causal alternatives to probing classifiers that intervene in model representations to examine effects on prediction also present another promising direction for future work @xcite @xcite . careful controls and baselines: we emphasize the need for probing work to establish careful controls and baselines when reporting experimental results. when probing accuracy for a linguistic competence is high, it may not be directly attributable to the training task. in this work we identify two confounds: incidental encoding and interaction between training tasks. perhaps future work will determine causes of incidental encoding and identify further baselines and controls that allow reliable conclusions to be drawn from probing studies. lack of gold-standard data of task requirements: while prior work has discussed the different linguistic competencies that might be needed for a task based on the results of probing studies, these claims are inherently hard to reliably quantify given that the exact linguistic competencies, as well as the extent to which they are required, are difficult to isolate for most real-world datasets. controlled test cases (such as those in §5.1) are effective as basic sanity checks for claims based on diagnostic classification, and provide insight into encoding dynamics in sentence representations. datasets are proxies for tasks, and proxies are imperfect reflections: finally, we speculate that while datasets are used as proxies for tasks, they might not reflect the full complexity of the task. aside from having dataset-specific idiosyncrasies in the form of unwanted biases and correlations, they might also not require the full range of competencies that we expect models to need to succeed on the task. future work should refine or move beyond the probing paradigm to carefully identify what the competencies reflected in any dataset are, and how representative they are of overall task requirements. what probes are good for: this work explores only the implications of probing as a diagnostic tool for pinpointing the information models use to do a task. however, when sentence representations are used subsequently downstream (after being trained on the main task), probing can give insight into what information is encoded in the model (irrespective of how that encoding came to be). future work could include exploring the connection between information encoded in the representation and whether models successfully learn to use them in downstream tasks. ## conclusion the probing paradigm has evoked considerable interest as a useful tool for model interpretability. in this work, we examine the utility of probing for providing insights into what information models rely on to do tasks, and requirements for tasks themselves. we identify several considerations when probing sentence representations, most strikingly that linguistic properties can be incidentally encoded even when not needed for a main task. this line of questioning highlights several fruitful areas for future research: how to successfully identify the set of linguistic competencies necessary for a dataset, and consequently how well any dataset meets task requirements, how to reliably identify the exact information models rely upon to make predictions, and how to draw connections between information encoded by a model and used by a model downstream.