Explore Spurious Correlations at the Concept Level in Language Models for Text Classification (2024)

Abstract: Language models (LMs) have achieved notable success in numerous NLP tasks, employing both fine-tuning and in-context learning (ICL) methods. While language models demonstrate exceptional performance, they face robustness challenges due to spurious correlations arising from imbalanced label distributions in training data or ICL exemplars. Previous research has primarily concentrated on word, phrase, and syntax features, neglecting the concept level, often due to the absence of concept labels and difficulty in identifying conceptual content in input texts. This paper introduces two main contributions. First, we employ ChatGPT to assign concept labels to texts, assessing concept bias in models during fine-tuning or ICL on test data. We find that LMs, when encountering spurious correlations between a concept and a label in training or prompts, resort to shortcuts for predictions. Second, we introduce a data rebalancing technique that incorporates ChatGPT-generated counterfactual data, thereby balancing label distribution and mitigating spurious correlations. Our method’s efficacy, surpassing traditional token removal approaches, is validated through extensive testing.

## introduction pre-trained language models (lms), leveraging extensive text corpora in their pre-training phase, have demonstrated remarkable effectiveness in a variety of natural language understanding tasks @xcite . nevertheless, lms encounter issues with spurious correlations during fine-tuning or instruction-following stages @xcite @xcite . these correlations involve specific associations between features and labels that, while prevalent in training data, are erroneously generalized as rules, leading to reduced performance. ## test sentence 0 the thai steak is overdressed and underwhelming. the french dip looked delicious. i know better for next time. ## dataset construction and analysis we fully fine-tune the distillbert. for llama2, we apply the lora method for efficient fine-tuning @xcite . details of the model implementations are included in appendix a. ## related work robustness and bias current work on studying spurious correlations for lms can be split into two categories: utilize the shortcuts during training or icl. for shortcut learning in training, a series of works explores how models take shortcuts in the data for the causal or non-causal perspective @xcite @xcite @xcite @xcite and which aspects of shortcuts will be taken for the predictions in different nlp tasks @xcite @xcite @xcite , leading to low generalization in the out-of-distribution data or in the designed adversarial data. due to the increasing development of llm on icl, researchers find that the design of the prompt significantly decides the llm predictions @xcite @xcite @xcite ). another line of work finds that llms are sensitive to a certain aspect of prompts and not robust when injecting adversarial triggers into prompt @xcite @xcite @xcite . @xcite shows that llms use multiple types of shortcuts in the prompts, from letters to words to text style, and @xcite find that llms exhibit clear feature biases under the unspecified prompts. previous work also develops multiple methods to identify the topic or concept of text input @xcite @xcite . however, our paper is the first to focus on assessing whether the models use shortcuts at the general concept level. data amazon shoe: size amazon shoe: color amazon shoe: style imdb: acting bias@c acc@noc acc@c bias@c acc@noc acc@c bias@c acc@noc acc@c bias@c acc@noc acc ## conclusions in this paper, we explore the spurious correlation at the general concept level in both fine-tuning and icl settings. we find that lms utilize the concept shortcut in training data (or in demonstrations) when inference on unseen data, and more biased training data (or prompts) lead to more biased predictions. to mitigate the learned shortcut, we propose a rebalancing method by adding counterfactual examples generated from chatgpt to the training data, shown to be effective through extensive experiments. our work indicates that lms form strong spurious correlations on general concepts, encouraging future work to pay attention to unintended shortcut learning.