Can Out-of-Distribution Evaluations Uncover Reliance on Prediction Shortcuts? A Case Study in Question Answering (2025)

Abstract: A large body of recent work assesses models’ generalization capabilities through the lens of performance on out-of-distribution (OOD) datasets. Despite their practicality, such evaluations build upon a strong assumption: that OOD evaluations can capture and reflect upon possible failures in a real-world deployment. In this work, we challenge this assumption and confront the results obtained from OOD evaluations with a set of specific failure modes documented in existing question-answering (QA) models, referred to as a reliance on spurious features or prediction shortcuts.We find that different datasets used for OOD evaluations in QA provide an estimate of models’ robustness to shortcuts that have a vastly different quality, some largely under-performing even a simple, in-distribution evaluation. We partially attribute this to the observation that spurious shortcuts are shared across ID+OOD datasets, but also find cases where a dataset’s quality for training and evaluation is largely disconnected. Our work underlines limitations of commonly-used OOD-based evaluations of generalization, and provides methodology and recommendations for evaluating generalization within and beyond QA more robustly.

## introduction improving the generalization of language models (lms), i.e., their capability to perform well beyond patterns covered by their limited training data @xcite , presents one of the most important challenges in modern nlp, with direct implications to their practical applicability in a wide variety of tasks. the most common approach towards evaluating lms' ability to generalize is to assess their performance on so-called ## background evaluating generalization in qa evaluations on out-of-distribution (ood) datasets @xcite are decisively the most popular method for evaluating generalization of language modelsto the point that performance on ood is often even referred interchangeably with the term of generalization @xcite . within qa, among many others, @xcite train models on squad @xcite as indistribution (id) and evaluate generalization on triviaqa @xcite , newsqa @xcite , searchqa @xcite or naturalquestions @xcite as ood. @xcite evaluates even shortcuteliminating method by training on squad and evaluating on adversarialqa @xcite as ood. @xcite train on squad as id and evaluate on triviaqa as ood. despite its known blindspots, squad still remains the default training dataset for a majority of the most popular qa models on huggingface. with a primary objective of improving generalizations, these works assume that ood evaluations can also uncover reliance on non-representative, spurious features. 1 in this work, we question this assumption and find cases where ood evaluations are largely 1 while we focus on qa, we can easily find such an assumption in other tasks, including nli @xcite or classification @xcite . prediction shortcuts a complementary yet less prevalent approach to assessing models' generalization aims to exploit functional failures identified in previous models. one approach towards this goal consists in identifying models' prediction shortcuts, i.e. a reliance on spurious features that are not representative for the learned task in general. such shortcuts were previously identified in nli @xcite , in-context learning @xcite or question answering @xcite . while these shortcuts are difficult to identify, with their knowledge, we can test the model specifically for the reliance on each of the shortcuts. this can be done by constructing synthetic data @xcite or subsetting existing data @xcite into subset(s) where we make sure that a specific shortcut is not applicable. in nli, where the impact of specific prediction shortcuts is widely studied, a common practice is to evaluate on datasets specifically constructed to exploit reliance on shortcuts @xcite . however, in open-ended tasks including qa or multitask benchmarks such as mmlu @xcite , the applicability of shortcuts across different datasets used for ood evaluations remains under-studied. ## methodology our goal is to uncover if, and to what extent can commonly-performed ood evaluations capture models' failures attributed to models' reliance on prediction shortcuts. we approach this by reproducing several ood evaluations used in previous work and comparing the results of these evaluations with models' measured sensitivity to prediction shortcuts representing previously reported failures modes. models for our assessments, we aim to pick a set of existing qa models that are the most widelyused in practice. towards this goal and with a minor preference for diversity, we pick five among the thirty most-downloaded models on huggingface foot_0 with the added criterion that they must have been trained on squad only, so as to ensure a fair comparison. additionally, to further enhance diversity, we complement this selection by training a set of our own models covering the most popular model families among qa, such that all other training parameters (e.g, early stopping, batch size, learning rate) are fixed across different model families. id & ood datasets consistent with previous work ( §2) and the most popular qa models, we use squad @xcite as our id dataset unless otherwise noted. to evaluate robustness, we pick five different ood datasets, all used in prior studies. we then use the performances on each of these datasets, as well as their average, to establish a first set of rankings for our models. prediction shortcuts we create a second set of model rankings by evaluating their reliance on five shortcuts identified in previous work on qa and attested across all models in our study (detailed in §a): shared words @xcite , questionwords distance @xcite , keywords match @xcite , answer length @xcite and entity match @xcite . to quantify the reliance of a given model on each of these shortcuts, we follow the methodology of @xcite . in practice, we (1) split the in-distribution validation dataset into two segments based on whether the example (i) can or (ii) can not be solved by the shortcut; (2) compute the accuracy on both segments; and (3) calculate the relative drop in accuracy between the segments. 3 metrics we assess the agreement between the ood evaluations and models' reliance on shortcuts by visualizing the ranking obtained by each of these features, and their mutual correlation. all our reported results employ the exact-match metric. foot_2 in figure 2 , we compare the different rankings via kendall τ pairwise correlations, a metric proportional to the number of pairwise swaps needed to transform one ranking into another. results reveal that there are two vastly different ood datasets: naturalquestions and searchqa. rankings according to these datasets have minimal correlation (τ < 0.4) with each of the shortcuts, but, as we find, also other with datasets. both datasets correlate with the averaged reliance on shortcuts substantially worse than the traditional id evaluations. on the other hand, some ood evaluations correlate with average shortcuts' ranking much better than id or even averaged ood performance - which suggests that ranking among a large set of ood evaluations, assumed as more robust in some of previous works @xcite need not provide a better assessment of robustness than an informed selection of a single ood dataset. ## conclusions this paper investigates a discrepancy between ood evaluations used in previous work in qa as a proxy for generalization, and previously documented failures to generalize identified as uncover reliances on prediction shortcuts. by ranking a set of popular qa models according to these two facets, we find that datasets previously used for ood evaluations vastly differ in their capacity to uncover shortcuts. we find that prediction shortcuts are to a large extent shared across datasets used for both training and evaluations of generalization. however, the quality of datasets in uncovering shortcuts is not proportional to their capacity to train more robust models, possibly due to dataset-specific features with a different impact on training and evaluation. we hope our results will inspire future work towards a more systematic selection of ood datasets and provide concrete recommendations for ood datasets selection in qa. crucially, our findings may also motivate future work in generalization within and beyond qa to restrain from overgeneralized conclusions based on demonstratedly limited ood benchmarks. this project has received funding from the european union's horizon europe research and innovation programme under grant agreement no. 101070350 and from uk research and innovation (ukri) under the uk government's horizon europe funding guarantee (grant number 10052546). the contents of this publication are the sole responsibility of its authors and do not necessarily reflect the opinion of the european union.