Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration (2025)

Abstract: Stance detection is critical for understanding the underlying position or attitude expressed toward a topic. Large language models (LLMs) have demonstrated significant advancements across various natural language processing tasks including stance detection, however, their performance in stance detection is limited by biases and spurious correlations inherent due to their data-driven nature. Our statistical experiment reveals that LLMs are prone to generate biased stances due to sentiment-stance spurious correlations and preference towards certain individuals and topics. Furthermore, the results demonstrate a strong negative correlation between stance bias and stance detection performance, underscoring the importance of mitigating bias to enhance the utility of LLMs in stance detection. Therefore, in this paper, we propose a Counterfactual Augmented Calibration Network (FACTUAL), which a novel calibration network is devised to calibrate potential bias in the stance prediction of LLMs. Further, to address the challenge of effectively learning bias representations and the difficulty in the generalizability of debiasing, we construct counterfactual augmented data. This approach enhances the calibration network, facilitating the debiasing and out-of-domain generalization. Experimental results on in-target and zero-shot stance detection tasks show that the proposed FACTUAL can effectively mitigate biases of LLMs, achieving state-of-the-art results.

## introduction stance detection aims at automatically identifying the author's opinionated standpoint or attitude (e.g., favor, against, or neutral) expressed in the content towards a specific target, topic, or proposition @xcite the first two authors contribute equally to this work. ## related work biases in large language models some studies @xcite have examined the biases existing in large language models (llms), these biases mainly include gender and religion @xcite , politics @xcite , and spurious correlations @xcite . the associated debiasing efforts are centered around retraining the language model with debiased samples @xcite . @xcite found that llms are vulnerable to option position changes in mcqs due to their inherent 'selection bias'. they perform debiasing by approximating the overall bias distribution. while based on our analysis in section 3, the bias distribution varies significantly across different stance detection samples, so this method is not applicable. ## mitigating bias with calibration given {x n , t n } n @xmath0, where x denotes the input text and t denotes the corresponding target, llms obtain the stance predictions y through the task instructions i for stance detection: p obs (y i |i; x i , t i ). inspired by @xcite , we believe that it can be deconstructed into the unbiased distribution p unbiased of the llms performing the stance detection task, and the bias distribution p bias formed by confounding c i : the confounding factor c i arises from the two types of biases analyzed earlier. it is important to note that it may be unaffected, influenced by a single bias, or impacted by both types of biases. specifically, p obs denotes the stance judgment text, which can be regarded as the outcome of word probability distribution following the argmax operation, derived from llms. we aim to estimate the unbiased stance distribution p unbiased . ## conclusion in this paper, we categorize the biases of llms in stance detection into two types from the perspective of causality and propose metrics to quantify these biases. then, we propose a counterfactual augmented calibration network, coined as factual. in which, a trainable calibration network and counterfactual data augmentation are explored to mitigate the biases of llms in stance detection. experimental results on intarget and zero-shot stance detection show that our factual can effectively reduce the bias of llms in stance detection and contribute to improved performance.