Can Edge Probing Tests Reveal Linguistic Knowledge in QA Models? (2022)

Abstract: There have been many efforts to try to understand what grammatical knowledge (e.g., ability to understand the part of speech of a token) is encoded in large pre-trained language models (LM). This is done through ‘Edge Probing’ (EP) tests: supervised classification tasks to predict the grammatical properties of a span (whether it has a particular part of speech) using only the token representations coming from the LM encoder. However, most NLP applications fine-tune these LM encoders for specific tasks. Here, we ask: if an LM is fine-tuned, does the encoding of linguistic information in it change, as measured by EP tests? Specifically, we focus on the task of Question Answering (QA) and conduct experiments on multiple datasets. We find that EP test results do not change significantly when the fine-tuned model performs well or in adversarial situations where the model is forced to learn wrong correlations. From a similar finding, some recent papers conclude that fine-tuning does not change linguistic knowledge in encoders but they do not provide an explanation. We find that EP models are susceptible to exploiting spurious correlations in the EP datasets. When this dataset bias is corrected, we do see an improvement in the EP test results as expected.

## introduction the encoding of linguistic information in large pretrained language models (lms) such as bert @xcite has become an active research topic in recent times. this encoding is usually measured by edge probing (ep) tasks @xcite . consider the sentence "the met is closing soon". the token 'met' is a noun (a museum and not a form of the verb 'meet'). the context words ('the', 'is') are the only signals for determining its part of speech. if a 'simple' (one or two layer mlp @xcite * work done at the university of copenhagen. if encoder e 1 performs better than encoder e 2 on an ep test, say, part-of-speech tagging, we say that e 1 has a better knowledge of part-of-speech than e 2 . for many nlp tasks, pre-trained lms (most commonly, bert) have emerged as standard encoders @xcite . these encoders are fine-tuned after adding task-specific layers on top. while probing tests on pre-trained encoders are quite popular, fine-tuned encoders are relatively under-explored (with notable exceptions of @xcite and @xcite . we aim to bridge this gap by probing fine-tuned models using question answering (qa) as a target task. qa is a complex nlu problem requiring the model to implicitly perform many reasoning steps, and fine-tuned models provide strong baselines for various qa datasets @xcite . our first research question is thus: rq1: does fine-tuning for qa tasks improve the encoding of linguistic skills in the encoders, when measured by existing ep tests? intuitively, dnn based qa models would require implicit knowledge of semantic roles (who did what to whom, when, and where), an understanding of the part of speech and entity boundaries (most answers are entities in the context), and anaphora resolution (entities in the context would refer to each other). indeed, prior works show how injecting knowledge about semantic roles @xcite and coreference @xcite in classical qa pipelines improves their performance. therefore, a fine-tuned qa model should implicitly acquire these linguistic skills. the qa layers in the fine-tuned models have much fewer parameters than the encoders ( §3), therefore, the encoders themselves are more likely to encode that grammatical knowledge. consequently, when these fine-tuned encoders are used for the srl (semantic role labeling), coref (coreference), pos (part-of-speech tagging), and ner (named entity recognition) ep tests, we would expect to see improvements over pre-trained lms. but we do not observe any such change ( §4). fine-tuning is generally performed on much less data than pre-training and the encoder weights might not change significantly. could that cause the ep test results to remain the same? if the encoder weights are kept fixed during fine-tuning, the performance in the target (qa) task drops 50-70% on all datasets. however, this frozen encoder has the same performance on the ep test as the original one. in other words, two encoders with a high difference in the target task performances have no discernible difference in the ep task performances. a possible explanation is that the qa models have no need to use the grammatical knowledge we are testing for. this motivates the second research question: rq2: does fine-tuning for qa tasks impart the linguistic skills necessary to perform qa in the encoders? to answer this, we create a qa dataset that requires a particular 'skill' @xcite : the knowledge of coreference resolution ( §5). quoref @xcite is such a dataset, but one might not require the knowledge of coreference to answer all questions in quoref. many instances in standard nlu tasks can be solved by heuristics, i.e., without proper reasoning (see @xcite for nli or @xcite for multi-hop qa). we design algorithms to filter out questions that can be answered heuristically ( §5), and consequently, any model probably needs to use the knowledge of coreference to answer the rest. however, two encoders with a significant performance difference on this de-biased dataset have no difference in the coref ep test. this motivates the third research question: rq3: why do the ep test results not reflect that encoders have learned the linguistic skills needed to perform qa?: our analysis ( §6) of the ep test datasets suggests that the ep models themselves might rely on dataset biases (as opposed to learning the task with input representations). when this bias is corrected, fine-tuned encoders behave as expected, i.e., show significant performance improvements over the base encoders. previously, van @xcite and @xcite observed that the ep test results do not differ in the base vs fine-tuned encoders foot_0 (rq1) and concluded that the encoding of grammatical knowledge in the encoders does not change during fine-tuning. however, unlike ours, their studies were not done on the problems that explicitly call for such grammatical knowledge. moreover, current criticisms of ep tests on non fine-tuned encoders focus on the task design itself @xcite ) (see §6), whereas this work calls for bias correction in the standardized ep test datasets. ## related work prior work has focused on understanding various aspects of pre-trained lms including attention patterns @xcite and linguistic knowledge @xcite . when these lms are used as encoders in models, they turn out to be strong baselines for many tasks @xcite . however, less is known about how the fine-tuning process changes the encoder's attention patterns @xcite or their encoding of linguistic knowledge. while many papers @xcite @xcite argue that dnn models often use heuristics to answer questions, ray @xcite shows that at least some of the models use human-interpretable reasoning steps. therefore, it is important to study how edge probing tests capture the task-specific reasoning abilities introduced in the fine-tuning process. the paradigm of the classifier based probing tasks (of which our ep tasks are a subset) is quite mature @xcite , and has seen increasing popularity with the release of benchmark ep datasets (the ones we use here) @xcite . typically, internal layers of large language or machine translation models are used as features for auxiliary prediction tasks for syntactic properties: part-of-speech @xcite @xcite , tense @xcite , or subject-verb agreement @xcite . see @xcite for an extensive survey. however, not many papers study the benchmark ep tests for fine-tuned representations. most similar to our work is the layer-wise analysis of bert weights for qa @xcite and the results of fine-tuning bert on ep tasks @xcite use three qa datasets (squad, hotpotqa, and babi @xcite ) to show that: 1) for the ep task of coref, test results remain unchanged, even when representations are taken from different layers; 2) different layers of a fine-tuned bert can be attributed to different tasks in the qa process such as supporting fact extraction or entity selection. @xcite studies mnli, dependency parsing, and qa (squad) to arrive at a similar finding, although their main results use a scalar mix of the weights from all layers of a fine-tuned bert (whereas our work uses the top layer). rq1 in our work can be considered complementary to their work, but rq2 and rq3 have not been studied before. ep tests are indirect, i.e., a classifier (probe) is used to measure the linguistic information in the representation. do the test results reflect the quality of the representations or the classifier's ability to learn the task @xcite ? we discuss this in §6. see @xcite for more background on probing classifiers. i eat strawberry ice cream pre-trained lm: (bert) t1 t2 t3 t4 t5 e1 s1 (t1) s2 (t2) s3 (t3) s4 (t3, 4, 5) e2 e3 e4 pos mlp (is s2 a vvp?) srl mlp (is s4 arg1 of s2?) original sentence encoder layer token representations from the encoder pooling layer on tokens pooled span representations: 1 or multiple tokens mlps for ep tasks (figure 1 ). for ep tests, the span representations need to be generated from the token representations. token representations can be generated from each layer @xcite or the top layer @xcite of the encoder. in each case, the layer i representation can be calculated as: 1. just the output of layer i, 2. a concatenation of the first layer output and layer i output ('cat'), and 3. a scalar mixing of the output of 0 -i layers where the mixing weights are trainable parameters ('mix'). in all experiments, we use the 'cat' setting for the topmost layer in the encoder because both @xcite and we find no significant difference for the other settings (the top layers generally perform better). question answering: in typical qa setups, models are given a context and a question as input. we use two span-based datasets @xcite and hotpotqa, @xcite ) where the task is to extract a span of the text from the context as the answer. we also use two mcq datasets @xcite and multirc, @xcite ) where the model is trained to select an answer from a set of choices. the architectures follow @xcite and are similar for all datasets: one or two qa-specific layers on top of an encoder (see the appendix for more details). ## rq1: ep tests & fine-tuned encoders we first run the ep tests with a standard encoder. next, the qa models (the same encoder + qa layers) are trained (fine-tuned). then, in the fine-tuned models the qa layers are replaced with the same mlp layers used in the ep tests, and the tests are repeated. this gives us a measure of how much the encoding of linguistic knowledge in the encoder might have improved due to the fine-tuning process. we also randomize the qa training data (for squad and hotpotqa) by using random noun phrases as answers. thus the model is forced to learn wrong correlations, which can cause it to 'forget' the skills to some extent, and in turn, the encoder should perform worse on the ep tests. ## rq2: ep test for coreference while we can expect the model to acquire some linguistic skills (the ability to do coreference resolution, identify the part of speech of a token) by learning to perform a qa task, there is no guarantee for this: a model can reason differently than we expect it to. for example, many hotpotqa questions can be answered by identifying the necessary entity type and not the multi-hop reasoning process that we expect @xcite . therefore, in rq2, we want to see whether the ep test results change when we know the encoder has to acquire particular grammatical knowledge k for the qa task. consider two models m 1 (e 1 + qa_layer 1 ) and m 2 (e 2 + qa_layer 2 ) in our encoder + qa layer architecture. assume we can identify a set of questions q that can only be answered using k. if m 1 performs significantly better than m 2 in these questions, we can say that e 1 has encoded more information about k than the e 2 because the qa layers are unlikely to encode that knowledge as they have much less parameters than the encoders. therefore, in the ep test for k, we can expect e 1 to perform better than e 2 . we define m 1 as a fine-tuned-encoder, where the full architecture (encoder (e 1 ) + qa layer) is trained; and m 2 as a frozen encoder: the encoder (e 2 ) is frozen and only the qa layer is trained. we choose k to be the grammatical knowledge of coreference. it is difficult to understand whether the knowledge of semantic roles or part of speech would be needed to answer a question, but it possibly can be done for coreference. for example, in figure 2a , it is relatively easy to see that to answer the question a human needs to resolve the reference 'he' in the second sentence to 'leo strauss'. ## rq3: an analysis of ep tests 6.1 analysis we see that the ep test results are surprisingly stable, even when we expect the fine-tuned encoder to learn or forget certain linguistic skills. ep tests are indirect measures of a representation's quality and have been criticized as such. @xcite shows that for some ep tests, a large pre-trained lm (elmo @xcite ) has the same performance as a random encoder. they conclude that the test measures the classifiers' ability to learn the ep task, and not the knowledge encoded in the representations itself. they propose using an information-theoretic (minimum description length or mdl) probe. in a similar vein, @xcite suggests designing a control task. a control task for an ep test is the same classification task, only the labels of the original task are changed so that it can not be recovered from the linguistic information. for example, two tokens with different partof-speech tags will be mapped to the same arbitrary label. a classifier (probe) that performs well on both the control task and the original ep test must be learning the correlations in the data, and not using any information from the representations. therefore, it can not measure the encoding of grammatical knowledge in the encoders. if an ep test has only two labels, they will just be flipped in the control task. this makes the control task and the original ep test the same problem for the probe, and they must have the same performance. therefore, for binary ep tests such as coref (the one we are interested in), no control task can be designed, but we use simple linear probes following @xcite 's recommendations. had we used mdl probes instead, would our conclusions in rq1 and rq2 change? prior work reports that the fine-tuned encoders do not show much difference in the mdl probes themselves @xcite . moreover, even in the original mdl probe paper, the coref test results are similar in a pre-trained vs random encoder @xcite . therefore, replacing the current ep tests with mdl probes should not change the findings for the previous research questions. does this conclusively mean that the linguistic skill is not improved in the encoder, even when the task calls for it? both @xcite and @xcite arrive at that conclusion, albeit without fine-tuning on a skill-specific target dataset such as quoref. we present a dataset bias explanation ( §6.2). note the ep test dataset is used in both @xcite , therefore the same explanation is valid for both these studies. ## conclusion and future work edge probing tests are the predominant method to probe for linguistic information in large language models. we use them to evaluate how the process of fine-tuning an lm for qa might change the grammatical knowledge in an encoder, and observe no significant differences between pre-trained and fine-tuned lms. more importantly, we find this phenomenon in carefully designed target tasks where the models must use the said grammatical knowledge. from similar ep test results, previous works have concluded that fine-tuning does not change the encoding of grammatical knowledge. however, our analysis provides a 'dataset bias' explanation for the consistency of the results and provides some clues as to why any representation tends to achieve very similar results for ep tests. this is different from the previous task-design criticisms of the ep tests. do fine-tuned nlu models score highly on benchmarks for the right reasons, i.e., follow the human reasoning process? this work shows some evidence in favor of that. the encoding of grammatical knowledge in qa encoders is improved as expected when the models are trained on the right datasets, and the dataset biases in the ep tests are corrected. in light of this evidence, in the future, we plan to identify the exact reasoning steps in the qa models through post-hoc explainability methods and study whether they align with the human reasoning steps.