On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations (2022)

Abstract: Recent work has shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to over-fitting and lack of generalization. To mitigate this problem, a common practice is to balance datasets by adding new instances or by filtering out “easy” instances (Sakaguchi et al., 2020), culminating in a recent proposal to eliminate single-word correlations altogether (Gardner et al., 2021). In this opinion paper, we identify that despite these efforts, increasingly-powerful models keep exploiting ever-smaller spurious correlations, and as a result even balancing all single-word features is insufficient for mitigating all of these correlations. In parallel, a truly balanced dataset may be bound to “throw the baby out with the bathwater” and miss important signal encoding common sense and world knowledge. We highlight several alternatives to dataset balancing, focusing on enhancing datasets with richer contexts, allowing models to abstain and interact with users, and turning from large-scale fine-tuning to zero- or few-shot setups.

## introduction effective human communication relies on our ability to understand extra-textual context based on common sense, world knowledge or shared cultural experiences, a property often cited as grice's second maxim of quantity: "do not make your contribution more informative than is required" @xcite . studies have estimated that only 12% of the information conveyed by text is mentioned explicitly @xcite . to illustrate this, consider the question "who is the president of the u.s.?". to answer it, a human reader is likely to presume many unstated propositions, as exemplified in tab. 1. in contrast to humans, supervised models often fail to generalize and understand implicit context, instead resorting to low-level correlations in who is the president of the u.s.? ## dataset-model arms race this section provides a view of recent research in nlp as an arms race between models and datasets. below we describe the conditions leading to this 1 we emphasize that balancing methods are still useful as they can lead to mitigation of some spurious correlations, and therefore better generalization @xcite , as well as potentially more efficient training. we argue that these methods are inherently limited in their ability to mitigate all spurious correlations. arms race, and present our main research question, challenging its value for making progress in nlp. models exploit spurious correlations while pretrained models consistently perform well across multiple tasks, various studies have pointed out that this is often achieved by exploiting spurious correlations in datasets, rather than improving on the underlying task @xcite @xcite , and that this phenomenon becomes more prominent as the models grow in size @xcite . ## the lost battle against spurious correlations so far we have identified dataset balancing as a common way to mitigate spurious correlations. next, we outline how different works define spurious correlations (sec. 3.1), and then question whether dataset balancing is a viable way for mitigating them; we note that balancing too little is bound to leave spurious correlations in the data (sec. 3.2), while balancing too much discards meaningful signal (sec. 3.3). we finish by questioning whether this practice is even desired (sec. 3.4). ## ways to move forward so far, we presented limitations of dataset balancing as a means to mitigate spurious correlations. in this section we discuss several alternatives to this practice, summarized in tab. 5. we note that none of these proposals is particularly novel. rather, we intend to survey alternatives proposed in literature and argue that these may be promising for addressing the drawbacks of spurious correlations, and that more efforts should be put into studying them. ## related work this paper discusses the arms-race between models and datasets. previous works criticized one side of this arms race-the increasing size of pretrained models-due to ethical and environmental concerns @xcite , or questioning its ability to learn meaningful abstractions from raw text @xcite . this work studies the second part of this arms race, regarding the efforts to mitigate spurious correlations through dataset balancing. the release of such datasets is often motivated by their potential to spur progress in modeling, and to help tease apart qualitative differences between models. @xcite showed that this is not necessarily the case, by observing that the ranking of reading comprehension models on small and synthetic benchmarks is similar to that of the (large) squad dataset @xcite . @xcite recently criticized the concept of benchmarks as a whole, arguing that they are only capturing specific skills and not "general" capabilities. our paper raises related concerns about training sets implicitly containing spurious correlations, and suggests reconsidering the practice of building large-scale training sets. finally, concurrent to this work, eisenstein (2022) discussed several types of spurious correlations in the context of causality theory @xcite , and used a toy example to demonstrate their different effects on models. they concluded that domain knowledge is required to identify the correlations that are indeed spurious, i.e., those that might challenge the generalization ability of models. ## conclusion spurious correlations in large textual corpora can result in model brittleness, lack of generalization, and an inflated sense of the state of the art. mitigating their negative side-effects is an important research goal of the nlp community. in this paper we presented practical and conceptual limitations of dataset balancing as a means for doing so. we proposed alternative ways for mitigating spurious correlations, including adding richer contexts to textual corpora, and allowing models to abstain or interact in cases of uncertainty. we concluded by suggesting to reconsider the practice of fine-tuning pretrained models on large-scale training sets.