Implications of Annotation Artifacts in Edge Probing Test Datasets (2023)

Abstract: Edge probing tests are classification tasks that test for grammatical knowledge encoded in token representations coming from contextual encoders such as large language models (LLMs). Many LLM encoders have shown high performance in EP tests, leading to conjectures about their ability to encode linguistic knowledge. However, a large body of research claims that the tests necessarily do not measure the LLM’s capacity to encode knowledge, but rather reflect the classifiers’ ability to learn the problem. Much of this criticism stems from the fact that often the classifiers have very similar accuracy when an LLM vs a random encoder is used. Consequently, several modifications to the tests have been suggested, including information theoretic probes. We show that commonly used edge probing test datasets have various biases including memorization. When these biases are removed, the LLM encoders do show a significant difference from the random ones, even with the simple non-information theoretic probes.

## introduction word embeddings generated from large corpora can be expected to encode knowledge about syntax and semantics @xcite . this is certainly truer for the contextual ones from large language models such as elmo @xcite , bert @xcite or roberta @xcite . edge probing (ep) tests @xcite are standard classification tasks to probe for such knowledge. consider the sentence "the met is closing soon", the word "met" functions as a noun, referring to a museum rather than the past form of the verb "meet". to determine its part of speech, humans rely on the context words "the" and "is". if a classifier predicts this token as a noun using only the representation from a contextual llm encoder such as bert (i.e., without using the entire sentence), it is implied that these contextual signals are encoded within the token representation itself. ep tests aim to uncover such syntactic and semantic knowledge encoded ( §2). ep tests are however indirect measures of such knowledge. a high accuracy of an encoder in an ep test for a grammatical property in itself does not necessarily guarantee that the said knowledge is encoded. instead, the score should be significantly higher than the same from a baseline, which is typically set as static embedding encoders @xcite or contextual encoders with random weights @xcite @xcite . nlp tasks are typically modeled by datasets, albeit imperfectly @xcite , and consequently, the performance of the encoders in the ep tests are confounded by the choice of the test dataset and its inherent biases. despite a long history of research in edge probing tests, this problem has not been studied well @xcite . to bridge this research gap, we propose three research questions. rq1: are there "annotation artifacts" in the ep test datasets? many standard nlp datasets have data points that can be solved by superficial cues, i.e., reasoning strategies unrelated to the expected causal mechanism of the task at hand @xcite . for example, @xcite show that a negation operator in the premise is a strong predictor of the "contradiction" class in the snli @xcite dataset. @xcite show that in popular extractive machine reading comprehension (mrc) datasets such as squad @xcite or hotpotqa @xcite , in many cases the answer phrase can be found in the first sentence of the context. we analyze 17 ep test datasets across 10 tasks and find different biases in multiple of them. rq2: do the ep models use heuristics? existence of annotation artifacts in the data does not necessarily imply that the models will learn to use the related heuristics, eg., predict "contradiction" whenever the premise contains a negation. we can a) remove the biased test data @xcite or b) adversarially perturb it @xcite and observe the performance degradation (if any) of a model. a significant degradation will indicate that the model does depend on the heuristic. using this technique, we show that the ep classifiers trained with random encoders do indeed learn to use the heuristics to a large extent, whereas the same ones trained with pre-trained encoders do not in the same capacity. rq3: do the pre-trained encoders encode grammatical knowledge better than the random encoders? a strong criticism of ep tests is that often the performances of the pre-trained and the random encoders are not significantly different @xcite . this is often attributed to the "classifier knowledge" problem, i.e., the ep classifier learns the task itself and does not necessarily depend on the encoder representations. various information theoretic probes @xcite have been proposed to solve this, including a popular one based on the minimum description length (mdl) principle @xcite . in this mdl probe @xcite , a combined measure defined on the ep classifier model complexity and its performance is minimized. the mdl codelengths of contextualized representations such as elmo are shown to be much lower than the corresponding random ones even when their ep test accuracies are very similar. however, we show this is not strictly necessary, and the similar performance of a pre-trained and random encoder can largely be attributed to the ep test dataset biases, as in when the "biased" data points are removed, a simple linear or mlp classifier shows a significant difference in the pre-trained vs random encoder. we investigate this further and show that bayesian classifiers such as mdl probes are not "inherently better" in testing an encoder's ability to encode grammatical knowledge. ## annotation artifacts in ep test datasets our analysis indicates that almost all ep test datasets have a significant repetition bias: many samples in the training data are repeated in the test. however, their labels may always not be the same, for example, in the ner ep test, the span "google" might have the label "org" or "o" depending on whether the span refers to the company or the search engine developed by it. we ask two questions. in a test dataset, in what percentage of cases a test data point is in the training data and has only one label? for example, in the ner datasets, if the span "google" appears in both the training and the test dataset with the only label "org", the ep classifier can successfully classify it by memorization. we call it the mem-exact heuristic. even if the training data contains multiple labels for a span (eg., both "org" and "o"), the ep classifier might be able to successfully classify it in the test data by simply learning the label distribution for the span and not the inherent contextual relationships. in the mem-freq heuristic we find the percentage of test data points that are present in the training data and can be classified correctly using the training label distribution. we also consider a baseline: the mem-uniform heuristic where instead of the true label distribution the class labels can be predicted by sampling from a uniform distribution. ## do the ep models use heuristics? based on the dataset biases discovered in §3, we hypothesize that the ep classifiers can use heuristic algorithms, but there will be a difference in the random vs pre-trained encoders. specifically, ep test classifiers with random encoders will learn to use various heuristics as the input representations themselves do not provide much information. on the other hand, the same classifier models with pretrained encoders will tend to not make use of such heuristic mechanisms. if the hypothesis is true, we will see a significant drop in the performance with the random encoders compared to the pre-trained encoders when the "heuristically classifiable" data points are removed from the test data. ## ep test results: random vs pre-trained encoders previously, we have shown that the random encoders show a significant memorization bias compared to the pre-trained ones. how does that affect the ep test results? in its original formulation, the minimum description length (mdl) principle is a bayesian model selection technique. a model class m is a set of models m i , for example, m can be "all polynomials of degree 3" and one m i can be 5x 3 . between two model classes m a and m b , the better model class is the one with the lower stochastic complexity. given a supervised classification dataset d with data points d @xmath0, y i ⟩, a model m defines a probability distribution p (y i |x i ). from the kraft-mcmillan inequality, there exists a code c for d with the code length naturally, a better model fit cor m i (with parameters θ i ) through maximum likelihood estimation that leads to the maximum p , hence the minimum code length l(d| θ(d)) = -logp (d| θ(d)). crucially, we are not allowed to fit a different θ and build a new code c ′ with each new dataset d ′ . ideally, we would like to have a single code c * that can yield the minimum length for all datasets but that is not possible if m contains more than one model. nevertheless, it is possible to construct c * such that: @xcite ) equation 1 is a combination of the "goodness of model fit" (better estimate of θ =⇒ smaller code length) and the model complexity (k * ). k * can be approximated for a regular model class m containing models with p parameters as: voita and titov (2020) follows blier and ollivier (2018) in determining code lengths for dnn models because the approximation in eq. ( 2 ) is not correct for complex dnns. but the ep classifiers are not dnns, they are simple linear models whose code lengths should be approximable by eq. ( 2 ). but as eq. ( 2 ) shows, the code lengths are not dependent on the datasets as long as the number of data points is large, which is true for most ep datasets (see ## related work previous research has primarily focused on studying different aspects of pre-trained language models (lms), such as linguistic knowledge @xcite and attention patterns @xcite . the paradigm of classifier-based probing tasks is well-researched @xcite and has gained popularity with the introduction of benchmark ep datasets that we utilize here @xcite . typically, internal layers of large language or machine translation models are used as features for auxiliary prediction tasks related to syntactic properties, such as part-of-speech @xcite @xcite , tense @xcite , or subjectverb agreement @xcite . for a comprehensive survey, refer to @xcite . ep tests are not direct evaluations of models since they use another model (called probe) to extract and evaluate the linguistic features within an encoding. because of this, it is not clear if the results reflect the quality of encoding or the probe's ability to learn the task @xcite @xcite . we delve into this topic further in section 5. additional details can be found in @xcite . ## conclusion ep tests are classification tasks to measure an llm's ability to encode syntactic and semantic knowledge. however, in many ep datasets, there is not a significant difference between the random vs pre-trained encoders, which raises questions about the validity of the tests (the "classifier knowledge" problem). we analyze 17 datasets across 10 datasets to find various biases and show that the ep classifiers are more prone to use heuristic mechanisms when random encoders are used instead of the pre-trained ones. when the dataset biases are removed, the pre-trained encoders do show a significant difference from the random ones as expected. information-theoretic probes have been proposed before to solve the "classifier knowledge" problem, we show why they might not be necessary. future work would extend the findings of this study to fine-tuned models.