Classifier Probes May Just Learn from Linear Context Features (2020)

Abstract: Classifiers trained on auxiliary probing tasks are a popular tool to analyze the representations learned by neural sentence encoders such as BERT and ELMo. While many authors are aware of the difficulty to distinguish between “extracting the linguistic structure encoded in the representations” and “learning the probing task,” the validity of probing methods calls for further research. Using a neighboring word identity prediction task, we show that the token embeddings learned by neural sentence encoders contain a significant amount of information about the exact linear context of the token, and hypothesize that, with such information, learning standard probing tasks may be feasible even without additional linguistic structure. We develop this hypothesis into a framework in which analysis efforts can be scrutinized and argue that, with current models and baselines, conclusions that representations contain linguistic structure are not well-founded. Current probing methodology, such as restricting the classifier’s expressiveness or using strong baselines, can help to better estimate the complexity of learning, but not build a foundation for speculations about the nature of the linguistic structure encoded in the learned representations.

## introduction the impressive performance of neural language models such as elmo @xcite and bert @xcite has led many authors to believe that these models must have learned linguistic structure, and a whole new research branch concerned with the analysis and interpretation of neural models has emerged, with dedicated areas at many conferences and the blackboxnlp workshop being established as a large and well-known venue in a short time. an appealing and widespread analysis technique, perhaps due to its simplicity and generalizability, is to use a model's word representations as input to a simple classifier, and train this classifier on an auxiliary task that can be expected to benefit from linguistic knowledge. the argument behind this technique is that, if a simple classifier is enough to learn the task based on word-level representations alone, then it is probable that these representations indeed encode the hypothesized linguistic structure @xcite . many of the papers applying classifier probes suggest that models such as bert and elmo encode structure that is similar to well-known syntactic annotations, e.g. dependency trees @xcite , or even to a traditional nlp pipeline @xcite . a standard assumption in probing is that, if we feed only the representation of one word at a time to the classifier, then no complex inference based on other words in the sentence can happen, and the probe will not be able to learn the classification task well-unless the necessary linguistic information already is encoded in the representation @xcite . therefore, the argument goes, high performance on a probing task can be taken as evidence that the word-level representations encode linguistic structure. in section 3, we challenge this view by showing experimentally that the word-level representations learned by elmo and bert encode the closer neighborhood of the word in a surprisingly exact way. to explore the consequences of this observation, we formulate the context-only hypothesis, which offers an alternative explanation of how a probe could reach high accuracy on the auxiliary task: it could simply make use of a compressed representation of the neighboring words. we propose that any work arguing to have found linguistic structure in word representations should be able to reject this hypothesis. in section 4, we examine whether past studies using classifier probes are capable of rejecting our context-only hypothesis, and find that they are not. to argue for the presence of linguistic structure in word embeddings, most of these studies either compare the probe's performance to a baseline, or restrict the probe's expressiveness or the number of training examples. regarding the first line of argument, we scrutinize proposed baselines and argue that none of them allows us to conclude the presence or absence of linguistic structure in embeddings. we posit that, as long as a model is not fully understood, such baselines probably cannot be found. regarding the second line, we argue that training restrictions do not provide a reliable setup either, as long as no theoretical ground exists that proves that a certain classifier with a certain set of training data is too weak to learn the probing task. we present experiments showing that the most restricted probes often fail to outperform even uncontextualized baselines. a high variation in very restricted setups also indicates that the results of experiments with restricted training scenarios are hard to interpret, and that their interpretation hence mandates careful testing and error analyses. we speculate that the correct interpretation of such experiments would probably also require full understanding of the model, as well as a clear definition of the distinction between "learning the task" and "extracting linguistic structure", which is lacking for dense embeddings. we conclude the paper by arguing for an alternative view of the results obtained with classifier probes, namely to regard word representations as containing features that are more or less helpful to learn the probing task. ## preliminaries we start the paper by introducing our notation and general experimental setup. ## extracting linguistic structure, one embedding at a time the purpose of a probe is to reveal what linguistic structure is encoded in word representations. there is little doubt that these representations contain useful information; but whether this information takes the form of traditional relational information between words, such as syntactic or semantic structure, is quite a different issue. ## a framework for the analysis of probing experiments the experiments that we presented in the previous section show that pre-trained word representations encode the sentential context of a token to a significant extent. but how can we know whether they encode any specific linguistic structure? in this section we provide a framework for the theoretical analysis of a probing experiment with regards to this question. ## conclusion being a very young field of research, probing and interpretation of language representations still lacks a solid theoretical foundation of its methodology, and despite some progress, the inner workings of models such as bert and elmo remain largely obscure. we have shown that a simple classifier trained on these models' representations can reconstruct the words in the context of single tokens with surprisingly high accuracy. based on this observation, we propose to evaluate probes with regard to the context-only hypothesis: that the probing classifier learns the task based on linear context information alone. with this hypothesis in mind, many attempts to discern "learning the probing task" from "extracting linguistic structure" are not reliable in theory and in practice. looking at random, uncontextualized and contextualized baselines, we have shown that none of them qualifies as a reliable ground for general conclusions about the linguistic knowledge that may be used to solve any sentence-level syntactic task. for the usage of probing model restrictions, we have argued that we are lacking too much knowledge about the representations and about their interplay with the probe's learning capabilities to reason about the presence of some specific feature from the results. a closer look at the learning curves evaluated at small numbers of training steps revealed that conclusions from probes with restricted training regimes have to be handled and interpreted with extreme care. in the most restricted setups, many of the probes fail to outperform even uncontextualized baselines in syntactic dependency pair and label prediction, and it remains theoretically and empirically unclear what setup to choose for a meaningful and reliable probe. in very recent work, @xcite take an information-theoretic perspective on probing, concluding that the difference between learning a task and identifying linguistic structure is non-existent. they argue against any restrictions on the expressiveness of the probe, stating that there is no information gain induced by them. we argue, in line with them, that valid conclusions using restricted probing classifiers can to this point only relate to the ease of learning with the features present in the representations. we propose that, as probing classifiers' simplicity is still appealing, future work should aim at providing theoretically well-motivated setups for this idea, as for example explored by @xcite and @xcite . we believe that approximating the ease of learning the task with the respective embeddings of the baseline and the representations in question e.g. with plots can be insightful to get an impression of the learning process, but interpretations remain difficult without an extensive error analysis, and comparisons of embeddings or data sets are especially hard due to differences in dimensionalities, numbers and distributions of labels.