Analyzing Biases to Spurious Correlations in Text Classification Tasks (2022)

Abstract: Machine learning systems have shown impressive performance across a range of natural language tasks. However, it has been hypothesized that these systems are prone to learning spurious correlations that may be present in the training data. Though these correlations will not impact in-domain performance, they are unlikely to generalize well to out-of-domain data, limiting the applicability of systems. This work examines this phenomenon on text classification tasks. Rather than artificially injecting features into the data, we demonstrate that real spurious correlations can be exploited by current state-of-the-art deep-learning systems. Specifically, we show that even when only ‘stop’ words are available at the input stage, it is possible to predict the class significantly better than random. Though it is shown that these stop words are not required for good in-domain performance, they can degrade the ability of the system to generalize well to out-of-domain data.

## introduction machine learning systems have shown impressive performance across a wide range of natural language processing (nlp) tasks such as questionanswering, sentiment classification and summarization @xcite @xcite . often these systems reach or even exceed human performance @xcite , which has led to increasing deployment of these automatic systems in real-world applications. there is, however, a caveat to the superhuman claim: standard benchmarks @xcite often assume that the training and evaluation data are drawn independently and identically from the same underlying distribution, an assumption that is rarely valid in 1 github repository: https://github.com/ adianliusie/stopword-bias the real world due to different deployment environments and constantly evolving evaluation distributions @xcite . high performance on the in-domain test set demonstrates that the system goes beyond memorization to successfully handle unseen examples. however this may only be true for a restricted domain, and hence the model may not generalize well to examples outside the training domain @xcite . an obstacle for generalization of machine learning systems is caused by the presence of spurious correlations. for example, in sentiment classification there may be a bias in the training data such that positive examples are longer than negative examples. in such scenarios, a model may use length as a significant feature to classify, which is problematic since length is 'a spurious feature' and should not provide sentiment information. although the model may still have good performance on the indomain test set (where this bias holds), reliance on this spurious feature may cost generalizability for real world out-of-domain (ood) settings as it distracts the system from learning the true underlying 'core' features of the task @xcite . biases have been studied in literature, where the focus is primarily on ensuring models don't use sensitive properties such as gender and race @xcite . in this work we are instead concerned with biases to other less sensitive spurious correlations. spurious correlations have been explored in nlp @xcite . many 'shortcuts' (spurious features with high in-domian correlation, geirhos et al. ( 2020 )) have been found for many nlp tasks: @xcite show that nlp models are prone to relying on spurious features provided they are easy to extract, @xcite show that neural models are able to complete sto-ries using only the final sentence, while @xcite show that clues left in the hypothesis are alone sufficient to achieve reasonable natural language inference performance. it is further shown that when such models are evaluated on adversarial data sets where the spurious correlations are eliminated @xcite @xcite , model performance drops drastically. this work diverges from the standard setup, and instead examines the susceptibility of models learning biases from innocuous, unimportant features. in particular, we explore the predictive abilities of 'stop' words such as 'and', 'of' and 'the' for a range of varying text classification tasks. we further explore whether models rely on such spurious correlations and make biased decisions in ood settings. ## spurious correlations spurious features have no causal relationship with the labels, but have strong correlations with the labels within a specific domain. more precisely, for input x and its corresponding label y, a model m aims to approximate the underlying distribution p(y|x) for all (x, y) ∈ d, where d is the entire input-output space of the task. typically, data is sampled from a restricted domain, d a ⊂ d. let f s (•) denote a spurious feature extractor. spurious features can be used effectively for prediction in the restricted domain d a (equation 1 ), but they have no causal link to the label in the general domain (equation 2 ) and so are ineffective for prediction. we focus on identifying real spurious features in nlp tasks with significant correlations with the labels. these spurious correlations will consequently lead to biases in trained models, which though valid in-domain, may compromise ood performance where the spurious correlations do not hold. ## conclusions this work investigates the influence of spurious biases in standard text classification tasks. it is established that the stop word distributions of the positive and negative classes are substantially different, and this acts as a significant bias for several tasks including sentiment classification and question-answering. in particular, after corrupting an input example to only retain the shuffled stop words, a standard transformer-based language model achieves reasonable performance across tasks despite no meaningful task-specific information. it is further demonstrated that language models pick up on the training data's stop word distribution bias. though, the spurious bias does not harm performance, when evaluated in-domain we observe that a randomly initialized transformer model maintains the spurious bias in ood settings too where the same stop word bias does not hold. hence, the learnt stop word bias from in-domain influences the predictions of the model in ood, leading to performance degradation. future work will investigate post-processing techniques to mitigate such spurious biases in deployed systems.