Mitigating Spurious Correlations via Counterfactual Contrastive Learning (2025)

Abstract: Identifying causal relationships rather than spurious correlations between words and class labels plays a crucial role in building robust text classifiers. Previous studies proposed using causal effects to distinguish words that are causally related to the sentiment, and then building robust text classifiers using words with high causal effects. However, we find that when a sentence has multiple causally related words simultaneously, the magnitude of causal effects will be significantly reduced, which limits the applicability of previous causal effect-based methods in distinguishing causally related words from spuriously correlated ones. To fill this gap, in this paper, we introduce both the probability of necessity (PN) and probability of sufficiency (PS), aiming to answer the counterfactual question that ‘if a sentence has a certain sentiment in the presence/absence of a word, would the sentiment change in the absence/presence of that word?’. Specifically, we first derive the identifiability of PN and PS under different sentiment monotonicities, and calibrate the estimation of PN and PS via the estimated average treatment effect. Finally, the robust text classifier is built by identifying the words with larger PN and PS as causally related words, and other words as spuriously correlated words, based on a contrastive learning approach name CPNS is proposed to achieve robust sentiment classification. Extensive experiments are conducted on public datasets to validate the effectiveness of our method.

## introduction distinguishing between spurious correlations and causal relationships in linguistics is crucial for building robust text classifiers @xcite @xcite . for example, in the movies dataset @xcite containing imdb movie reviews, and is found to have a stronger correlation with positive sentiment than excellent @xcite . however, from the semantics, it should be excellent instead of and that causes a positive sentiment of a movie review, and the word and itself does not necessarily affect the review's sentiment. this motivates the construction of robust text classifiers by identifying and using words that are causally related to sentiment rather than spurious correlated ones @xcite . to identify words that are causally related to the sentiment, previous methods propose to consider a specific word as the treatment word and estimate the causal effect on the class labels, whereas sentences containing the specific word are considered as belonging to the treatment group and otherwise to the control group. causal effect estimation methods include text or propensity matching @xcite , augmented inverse propensity weighting (aipw) @xcite , and representation learning-based methods @xcite @xcite . there are also methods relaxing the common assumptions in complex scenarios @xcite @xcite . however, a critical issue when using causal effects to identify causally related words is that when multiple causally related words appear in the same sentence, the causal effect of each causal word on sentiment drops dramatically, making it difficult to identify these words. for example, consider a sentence with positive sentiment -this movie is excellent and marvelous. when estimating the causal effect of word excellent on sentiment, the matched sentences without the word excellent may be -this movie is @xcite and marvelous, in which [token] is a word other than excellent, and this sentence may also be recognized as positive sentiment. therefore, the causal effect of word excellent on the sentence sentiment will be small because other positive words (e.g., marvelous) also appear in the sentence. this poses a great challenge to the effectiveness of previous methods of identifying causally related words by comparing the causal effects of different words on sentence sentiment. to empirically reveal the limitations of exploiting average treatment effects (ates) to identify causally related words, we compute the average ate of positive and negative sentiment words as treatments on the kindle dataset @xcite . as shown in to fill this gap, we aim to answer the counterfactual question, i.e., the highest level in the causal ladder @xcite , 'if a sentence has a certain sentiment in the presence/absence of a word, would the sentiment change in the absence/presence of that word?', instead of the interventional question as in the previous studies, i.e., the second level in the causal ladder. we introduce both the probability of necessity (pn) and probability of suffi-ciency (ps) @xcite and theoretically derive the identifiability results of pn and ps under different sentiment monotonicities. we further propose a novel robust text classification approach, as shown in figure 1 , in which the signs of the estimated ates correspond to different sentiment monotonicities, and words with the lowest estimated pn and ps are considered as spuriously correlated words and thus removed to achieve robust text classification. extensive experiments are conducted on three public datasets, demonstrating the superiority of our proposal on both spurious correlated word identification and robust text classification. the contributions can be summarized as follows. ## experiments in this section, we conduct extensive experiments on our proposed method, aiming to answer the following research questions (rqs): ## conclusion this paper proposes a novel method for distinguishing causally related and spuriously correlated words in sentiment classification by leveraging the probability of necessity (pn) and the probability of sufficiency (ps), aiming to eliminate linguistic spurious correlations. theoretically, we derive the identifiability of pn and ps under different sentiment monotonicity assumptions. empirically, we conduct extensive experiments across multiple datasets and backbone models, covering both crossdomain and in-domain scenarios, to demonstrate the effectiveness of our method in causal word identification and sentence sentiment classification.