Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models (2024)

Abstract: While the Large Language Models (LLMs) dominate a majority of language understanding tasks, previous work shows that some of these results are supported by modelling spurious correlations of training datasets. Authors commonly assess model robustness by evaluating their models on out-of-distribution (OOD) datasets of the same task, but these datasets might share the bias of the training dataset. We propose a simple method for measuring a scale of models’ reliance on any identified spurious feature and assess the robustness towards a large set of known and newly found prediction biases for various pre-trained models and debiasing methods in Question Answering (QA). We find that the reported OOD gains of debiasing methods can not be explained by mitigated reliance on biased features, suggesting that biases are shared among different QA datasets. We further evidence this by measuring that performance of OOD models depends on bias features comparably to the ID model. Our findings motivate future work to refine the reports of LLMs’ robustness to a level of known spurious features.

## introduction unsupervised pre-training and vast parametrization @xcite enable large language models (llms) to reach close-to-human accuracy on complex downstream tasks such as natural language inference, sentiment analysis, or question answering. however, previous work shows that these outstanding results can partially be attributed to models' reliance on non-representative patterns in training data shared with the test set, such as the high lexical intersection of the entailed hypothesis to premise (tu et al., * first two authors contributed equally figure 1 : we quantify model reliance on a spurious feature using bootstrapped evaluation on segments of data separated by exploiting chosen bias (left) and subsequently, by measuring the difference in model's performance over these two groups (right), that we refer to as prediction bias ( §3). 2020) in natural language inference (nli) or the intersection of the question and answer vocabulary @xcite in extractive question answering (qa). a primary motivation for mitigating models' reliance on such features is to enhance their robustness in practice, avoiding fragility to systematic errors when responding the open-ended user requests. models' robustness is commonly assessed by measuring prediction quality on samples from other outof-distribution (ood) datasets @xcite @xcite . however, the ood datasets might share training biases introduced by shared features, such as data collection methodology, or human annotators' background @xcite . in such cases, conversely, a model reliant on biased correlations can reach higher ood score despite being more fragile to the adversarial inputs exploiting the biased correlation. with this motivation, we propose a framework to evaluate models' reliance on a biased feature in prediction by splitting evaluation data to two groups based on a biased feature and comparing the prediction quality on these two groups (fig. 1 ). this way, we assess a reliance on bias of diverse qa models for several previously and newly identified bias features identified in this work. finally, we assess the efficiency of the state-of-the-art debiasing methods in mitigating reliance on spurious features over a resampling baseline and compare the findings to the commonly assessed ood performance. we find that avoiding reliance on spurious features does not imply improvements in ood performance; in many cases, debiasing methods mitigate the model's prediction bias, but the ood performance drops, while counterintuitively, a magnification of bias reliance can also bring large ood gains. aiming to explain this, we directly evaluate the prediction bias of models trained on different datasets and confirm that even models trained on ood datasets often rely on the same spurious correlations comparably to the id models. this finding motivates the presented assessment of model robustness towards known biases, in addition to ood performance. this paper is structured as follows. section 2 overviews data biases observed in nlp datasets, recent debiasing methods, and the previous methods related to measuring inclination to spurious correlations. section 3 presents our method for measuring the significance of specific biases. we follow in section 4 with details on our evaluation setup, including the tested debiasing methods, addressed bias features, and the design of a set of heuristics that can exploit them. subsequently, in section 5, we measure and report models' robustness to biases and ood datasets before and after applying the selected debiasing methods and wrap up our observations in sections 6 and 7. problem definition given a set of inputs @xmath0..i with corresponding labels @xmath1..i from a dataset d id , a model m learns a task t by identifying features f 1..n that map each x j to a corresponding y j , assuming that the learned features must be consistent with d id . since the learned f 1..n are distributed in m and can not be directly evaluated, we assess whether the learned features are robust for the task t by evaluating m on samples x ood of the same task, but drawn from d ood ̸ ≈ d id ; we assume that if f 1..@xmath2, the model will also perform well on x ood . however, the consistency of the learned f k with both x id and x ood is merely a necessary and not a sufficient condition for f k to be robust; if there exists a pair (x, y) such that the pair is a valid sample of the task t , but is not consistent with f k , we denote f k as spurious or bias features for t and refer to models' reliance on such features as prediction bias. in natural language inference (nli), where the task is to decide whether a pair of sentences entail one another, @xcite identify llms' reliance on a lexical overlap and on specific shared syntactic units such as the constituents in the processed sentence pair. @xcite identify the model's sensitivity to meaning-invariant structure permutations. similarly, chaves and richter (2021) identify bert's reliance on the invariant morpho-syntactic composition of the input. in question answering, llms often rely on the positional relation of the question and possible answer words, such as assuming their close proximity @xcite . @xcite find that models tend to assume that questions and answers contain similar keywords, remaining vulnerable to samples with none or multiple occurrences of the keywords in the context. @xcite show models' preference for the answers in the first two sentences of the context, being statistically most likely to answer human-curated questions. a perspective direction circumventing the biases introduced in data collection is presented in adversarial data collection @xcite where the annotators collect the dataset with the intention of fooling the likelybiased model, possibly enhancing the model-in-theloop in several fine-tuning iterations. still, some doubts remain, as other work provides evidence that models trained on adversarial data may work better on adversarial datasets but underperform on other datasets @xcite , or introduce its own set of biases @xcite . nevertheless, our experiments ( §5.2) show that training models on an adversarially-collected adversarialqa dataset turns out to be among the most effective approaches to mitigating known prediction biases in question answering. debiasing methods a well-established line of work proposes to address the known dataset biases in the training process. karimi mahabadi et al. (2020) and @xcite obtain a more robust, debiased model by (i) training a biased model that exploits the unwanted bias, followed by (ii) training the debiased model as a complement to the biased one in a product-of-experts (poe) framework @xcite . @xcite extend this framework in the learnedmixin method, learning to weigh the contribution of the biased and debiased model in the complementary ensemble. @xcite simulate the model for non-biased, out-of-distribution dataset through counterfactual reasoning @xcite and use the resulting distribution for distilling target @xcite , similarly to the learnedmixin. biased samples can also be identified in other ways, for instance, by the model's overconfidence @xcite . in a complement to poe approaches, other works apply model confidence regularization on the samples denoted as biased. @xcite and utama et al. (2020a) downweigh the predicted probability of the examples marked as biased by humans or a model. @xcite find that a more precise calibration of the bias-detection model might bring further benefits to this framework, consistently with our observations ( §6). distributionally robust optimization (dro) methods are another group of reweighting algorithms, addressing assumed imperfection of training datasets by (i) segmenting data into groups of diverse covariate shifts @xcite and (ii) minimizing the worstcase risk over all groups @xcite . we note that our bias measurement method closely relates to group dro methods and can, for instance, serve as a method for quantifying per-group risk. robustness measures most of the work on enhancing models' robustness evaluates the acquired robustness on ood datasets. in some cases, the evaluation utilizes datasets specially constructed to exploit the biases typical for a given task, such as hans @xcite for nli, paws @xcite for paraphrase identification, or adversarialqa @xcite for question answering, that we also use in evaluations. similar to us, some previous work quantified dataset biases by splitting data into two subsets, comparing model behaviour between these groups. @xcite perform such evaluation over mnli, demonstrating large margins in accuracy func measure_bias(m, x, h, t h ): algorithm 1: we measure prediction bias of the model m exploited by the heuristic h on dataset x, as a difference of m 's performance on two groups (x 1 and x 2 ) obtained by segmenting the samples of x by the attribute a @xmath3. we bootstrap both evaluations, (@xmath4, @xmath5, and obtain two sets of measurements (e 1 and e 2 ), of which we subtract the upper and lower quantiles e ↑ and e ↓ (q ↑ = 0.975, q ↓ = 0.025) and consider such distance a scale of the learned prediction bias. over the two groups and superior robustness of bert over previous models. similarly, utama et al. (2020b) compare two groups based on prediction confidence. our prediction bias measure follows a similar approach in qa but provides a more reliable assessment thanks to bootstrapping. further, compared to the previous work, we assess models' reliance on a range of 7 spurious features, making our overall conclusions more robust. an ability to measure a model's reliance on undesired features is also applicable in quantifying socially problematic biases. previous work also utilizes specialized domain knowledge in models' bias evaluation but might not scale to other bias features; @xcite collect ambiguous contexts and assess the models' inclination to utilize stereotypes as prediction features. @xcite quantify lms' gender bias by the co-occurrence of selected gender-associated words with gender-ambiguous words, such as doctor. ## measuring prediction bias we assess a model's sensitivity to a known spurious feature in the following sequence of steps. this methodology is visualized in figure 1 , described in algorithm 1 and can be used to measure biases of any other qa model using the project repository 1 . we start by (i) implementing a heuristic, i.e. a method h : @xmath6, that for all samples of dataset x computes an attribute a @xmath7, yet predictive for our training set and (ii) we compute h(x) for each sample x of evaluation dataset x. (iii) we choose a threshold t h that we use to (iv) split the dataset into two segments by a h . finally, (v) we evaluate the assessed model m on both of these segments, in our case using exact match evaluation, and (vi) measure model prediction bias as the difference in performance between these two groups. using bootstrapped evaluation, we mitigate the effect of randomness by only comparing selected quantiles of confidence intervals. we propose to perform a hyperparameter search for the heuristic's threshold t h that maximizes the measured distance. interpretation given the reliance on bootstrapping, we state that the model's true performance polarisation is 0.975 × 0.975 = 95.06%-likely to be equal or higher than the measured prediction bias (with q ↑ = 0.975, q ↓ = 0.025 as in algorithm 1). nevertheless, one should note that the proposed measure should not be used in a standalone but rather in a complement to an id evaluation, as one can reduce the prediction bias merely by lowering the performance on the better-performing id subset. therefore, we report the values of prediction bias together with the performance on a worse-performing, i.e. presumably non-biased split. another consideration concerns the "natural" polarisation of difficulty between samples; that is a portion of prediction bias which can be explained by the features f that are representative for the evaluated task ( §1). one should note that the reduction of prediction bias is meaningful only down to the level of the natural sample difficulty. the validation set of squad contains the annotations by three annotators that we use to quantify a level of prediction bias that can be explained by the questions' natural difficulty (further denoted as human model); we report the minimum over prediction biases of the annotators among each other. finally, even though we perform a hyperparameter search for optimal heuristics' thresholds t h feasible for a given size of dataset splits, there are no guarantees on the maximality of the found 1 https://github.com/mir-mu/isbiased t h . hence, prediction bias only provides the lower bounds of the model's polarisation. ## experiments our main objective is to assess the efficiency of different training decisions in mitigating the reliance of the model on spurious correlations that can be present in datasets. in question answering, previous work identifies several spurious covariates in the squad dataset @xcite ; we build upon these findings and further extend the list of covariates learnable from squad. for each suspected bias feature, we first describe and implement the exploiting heuristics that we use to segment groups in the prediction bias measure ( §4.1). subsequently, we observe the impact of the selected pre-training strategies ( §4.2) and debiasing methods designed to address the over-reliance on biased features ( §4.3 - §4.4) on the prediction bias and ood performance of the resulting models. ## discussion pre-training and models' robustness the biaslevel analyses of diverse pre-trained models ( §5.1) suggest that the mere increase of pre-training data and model parameters guide the fine-tuned models to lower reliance on biased features. however, we can find exceptions, such as in the case of roberta-large and electra-base on anslen. we speculate that even larger volumes of data might make the model more attracted to taking a shortcut through easier problem formulations, such as through named entity recognition (cf. bert-base and roberta-base on sim-ents). comparing the prediction bias of in-context learners with the fine-tuned models, we see that multi-task learning does not necessarily result in lower prediction bias or increased performance in the harder group; while flan-t5 on average reduces bias almost to the human level, t0's quality is affected by spurious features even more than the models fine-tuned on biased squad. ## conclusion our work sets out to investigate the impact of various training decisions, including different pretraining and debiasing strategies, on models' reliance on specific spurious features in qa, complementing the commonly used out-of-distribution evaluations. we use squad to survey the existing and to identify new biased features but evaluate the reliance on these features for models trained on four different qa datasets. we find that (i) the ood performance of different base models usually corresponds to models' reliance on bias features. however, (ii) the stateof-the-art debiasing methods can improve ood performance without minimizing the model's reliance on spurious features, suggesting that dataset biases might be shared among qa datasets. (iii) we further evidence this by measuring the reliance on a spurious feature of models trained on other (ood) datasets and find ood models similarly or even more reliant on spurious features learnt from squad. we hope that our analyses will motivate future work to assess models' robustness also on a more detailed level of specific bias features, evading false conclusions on models' robustness, and, ultimately, accelerating progress towards creating more robust and reliable language models.