Counterfactual Debiasing for Fact Verification (2023)

Abstract: Fact verification aims to automatically judge the veracity of a claim according to several pieces of evidence. Due to the manual construction of datasets, spurious correlations between claim patterns and its veracity (i.e., biases) inevitably exist. Recent studies show that models usually learn such biases instead of understanding the semantic relationship between the claim and evidence. Existing debiasing works can be roughly divided into data-augmentation-based and weight-regularization-based pipeline, where the former is inflexible and the latter relies on the uncertain output on the training stage. Unlike previous works, we propose a novel method from a counterfactual view, namely CLEVER, which is augmentation-free and mitigates biases on the inference stage. Specifically, we train a claim-evidence fusion model and a claim-only model independently. Then, we obtain the final prediction via subtracting output of the claim-only model from output of the claim-evidence fusion model, which counteracts biases in two outputs so that the unbiased part is highlighted. Comprehensive experiments on several datasets have demonstrated the effectiveness of CLEVER.

## introduction unverified claims have been prevalent online with the dramatic increase of information, which poses a threat to public security over various domains, e.g., public health (naeem and bhatti, 2020), politics @xcite , and economics @xcite . therefore, fact verification, which aims to automatically predict the veracity of claims based on several collected evidence, has attracted lots of research interests @xcite @xcite @xcite . existing fact-checking datasets inevitably involve some biases since they are manually collected. for example, @xcite discover that negation words in claims are highly-correlated with the label 'refutes' in the fever dataset @xcite . such biases may mislead models to explore the spurious correlation between claim patterns and its label without looking into the evidence. in consequence, though models achieve promising performance on biased datasets, they suffer from obvious performance decline on outof-domain unbiased datasets and are vulnerable to adversarial attacks @xcite . to alleviate the aforementioned problems, several debiasing methods have been proposed, which can be mainly grouped into two categories. the first pipeline is based on data augmentation, which utilizes manually-designed schemes, such as word swapping @xcite and span replacement @xcite to generate additional data for training. however, these methods heavily rely on the quality of augmented data and are difficult to be employed under complicated circumstance, e.g., multi-hop evidence reasoning, due to their inflexible augmentation rules. the second pipeline aims to downweigh the contribution of biased samples to the training loss of main model, whose inputs are both claim and evidence. then, the key issue is how to recognize the biased instances. specifically, @xcite downweigh the claim involving n-grams that share spurious correlation with labels. @xcite assume instances correctly classified by the bias-only model are biased, where the input of bias-only model is the claim only. nevertheless, the former lacks the generalization to different types of biases since they only focus on n-grams; the latter relies on the assumption that the outputs of main model and bias-only model regarding the biased instances are similar, which does not always hold @xcite . moreover, the inaccurate and unstable outputs of bias-only model during training may mistakenly result in downweighing unbiased samples @xcite . unlike existing works based on augmentation or adjusting the data contribution on the training stage, we propose a novel method from a counterfactual view for debiasing fact verification, namely clever, which is augmentation-free and alleviates biases on the inference stage. in general, existing methods fuse the claim and the evidence to make the final prediction, which is equivalent to asking the model to answer a factual question: what will the output be if the model receives a claim and its corresponding evidence? causally, the total causal effect is estimated in this condition, where the output is affected by both the biases in the claim and the claim-evidence interaction information (see the causal graph in figure 1 ). in other words, claim biases are entangled with the claim-evidence fused information, making them difficult to be mitigated precisely and thus resulting in a biased output. to overcome this, we aim to obtain the debiased output by removing claim biases from the total causal effect. inspired by the progress of counterfactual inference @xcite , we would expect to ask a counterfactual question: what would the output be if the model only received a claim? that is, from a causal perspective, requiring the fact-checking model to learn the direct claim effect solely affected by claim biases. practically, we first train a claim-evidence fusion model and a claim-only model independently to capture the total causal effect and the direct claim effect, respectively. then, we subtract the direct claim effect from the total causal effect on the inference stage to obtain the total indirect effect, which is the final debiased prediction. taking figure 1 as an example, the claim is spuriously correlated with the false label 'refutes' due to the phrase 'did not'. therefore, the direct claim effect inclines to the label 'refutes' since it is affected by the claim only. however, though the probability of wrong prediction 'refutes' in total causal effect is still the largest, the prediction is turned towards the ground-truth label 'supports' via using the total indirect effect as the final output, where the high probability of 're-futes' induced by claim biases is counteracted. as biases have been mitigated, the total indirect ef-fect reflects the intrinsic claim-evidence interaction information, leading to an unbiased prediction. overall, the main contributions can be summarized as follows: ## related work in this section, we briefly review the related literature in both domains of fact verification and debiasing strategy. ## method in this section, we introduce the proposed debiasing framework clever in detail. firstly, we provide some background information of fact verification. then, we describe the method from a causal view. finally, we elaborate the detail of training and inference. the overview of clever is shown in figure 2 . ## experiments in this section, we conduct both quantitative and qualitative experiments on several public datasets to demonstrate the effectiveness of our proposed method clever. ## conclusion in this paper, we have proposed a novel counterfactual framework clever for debiasing factchecking models. existing works mainly follow the data augmentation pipeline and the weight regularization pipeline. unlike them, clever is augmentation-free and mitigates biases on inference stage. in clever, the claim-evidence fusion model and the claim-only model are independently trained to capture the corresponding information. on the inference stage, based on the potential outcome model in the domain of causal inference, a simple subtraction scheme is proposed to mitigate biases. comprehensive quantified and qualified experiments have demonstrated the superiority of clever.