EXPLAIN , EDIT , GENERATE : Rationale-Sensitive Counterfactual Data Augmentation for Multi-hop Fact Verification (2023)

Abstract: Automatic multi-hop fact verification task has gained significant attention in recent years. Despite impressive results, these well-designed models perform poorly on out-of-domain data. One possible solution is to augment the training data with counterfactuals, which are generated by minimally altering the causal features of the original data. However, current counterfactual data augmentation techniques fail to handle multi-hop fact verification due to their incapability to preserve the complex logical relationships within multiple correlated texts. In this paper, we overcome this limitation by developing a rationale-sensitive method to generate linguistically diverse and label-flipping counterfactuals while preserving logical relationships. In specific, the diverse and fluent counterfactuals are generated via an Explain-Edit-Generate architecture. Moreover, the checking and filtering modules are proposed to regularize the counterfactual data with logical relations and flipped labels. Experimental results show that the proposed approach outperforms the SOTA baselines and can generate linguistically diverse counterfactual data without disrupting their logical relationships.

## introduction multi-hop fact verification task, which discerns the truth from falsehood based on multiple hops of reliable evidence, becomes crucial in countering misinformation and counterfeit news spread on current social media platforms @xcite , especially in some specific domains such as politics @xcite , science @xcite and public health @xcite . however, many recent works often perform poorly under the multitude of distribution shifts due to an over-reliance on spurious correlations between input text and labels @xcite @xcite . it can potentially be addressed by counterfactual data augmentation (cda), using counterfactual instances generated by perturbing causal features within the input @xcite . several works have revealed that training with counterfactual data enhances the capability of the model to identify causal features and diminish its reliance on spurious correlations between the input text and the label, thus resulting in the improvement in out-of-domain (ood) generalization @xcite . in this paper, we seek to generate counterfactuals for multi-hop fact verification, instead of exploring the causal bias for a specific model. however, due to the complex logical relationships within the multi-hop input texts, developing such an approach poses some significant challenges. as shown in the first row of for multi-hop fact verification, as shown in the third row of to address these challenges, we develop a novel pipeline method, race (rationale-sensitive counterfactual generation), by focusing on the causal features within the rationales extracted from the multi-hop evidence using an explainability method. in specific, for each original instance, the explainer and editor modules are employed to produce the counterfactual evidence that logically corresponds to -but factually distinct from -the original claim. then, according to the counterfactual evidence, an entity-aware generator generates the counterfactual claims by synthesizing the semantic information across multi-hop evidence. during the above process, the checking and filtering modules are used to regularize the reasonableness of the output of each module from different aspects, resulting in fully labeled examples that can be used directly to augment the training data. the motivation here is that these rationales provide the intrinsic semantic and relational information for inferring its label, and present the factual consistency with its claim @xcite . it should be pointed out that race requires no external knowledge as used in @xcite besides the original training data, and is able to generate linguistically diverse and label-flipping counterfactuals while preserving logical relationships. compared to alternative approaches (e.g., chatgpt (openai, 2022)) ( § 4), training on the counterfactuals generated by race reveals the improvement in performance under different settings ( § 5.1), including in-domain, out-of-domain @xcite , and challenge settings @xcite . in addition, the intrinsic evaluation shows that the counterfactual claims generated by race are more logical and linguistically diverse than those produced by the baselines ( § 5.3, § 5.4). finally, we compare the results based on different generation models with baselines, illustrating that our method is generation model-agnostic ( § 5.5). ## related works debiasing fact verification a variety of advanced multi-hop fact verification methods have recently emerged in various domains due to the development of pre-trained models @xcite . nevertheless, most models exhibit poor ood generalization, primarily due to their over-reliance on spurious correlations between inputs and labels @xcite @xcite . thus, several works focus on the debiasing of fact verification models. @xcite have identified strong cues for predicting labels solely based on the claim. @xcite proposed an entity debiasing framework that mitigates entity bias from a cause-effect perspective. @xcite addressed the debiasing of fact verification models by augmenting the data with contrastive instances. atanasova et al. (2022) explored what information is sufficient to verify a claim, and proposed a cda schema for learning of (in)sufficient information. counterfactual data augmentation there is a growing academic interest in cda to improve model robustness. initial studies focus on humancrafted counterfactuals @xcite . recently, numerous automatic cda methods have been proposed for sentiment analysis @xcite @xcite , question answering @xcite , and natural language inference @xcite . however, these methods are primarily targeted to nlp tasks without requiring complex reasoning about the input. thus, their direct application to the multi-hop fact verification task presents considerable challenges. ## methodology given a claim c with its associated evidence @xmath0, e 2 , . . . , e n ), the aim of multi-hop fact verification is to infer whether the claim is supported or refuted by the evidence. we denote an instance in the dataset d as a triplet (c, e, y), where @xmath1, ref } is the verification label. the goal of race is to generate counterfactual data (c ′ , e, y ′ ) or (c, e ′ , y ′ ) that differ in some meaningful way from the original instance (c, e, y), where y ′ ̸ = y, c ′ and e ′ denote the counterfactual claim and counterfactual evidence, respectively. this setting poses some unique challenges, such as requiring to identify the causal features to be edited, ensuring sound logical relations in evidence editing and claim generation, and avoiding unverifiable claims. meanwhile, ensuring the semantic diversity and the minimal perturbation of the counterfactuals can also be challenging. to this end, we propose a general pipeline, race, to tackle these challenges. as shown in figure 1 , our race consists of four stages: (i) explainer: rationale extraction ( §3.1), (ii) editor: evidence editing ( §3.2), (iii) generator: claim generation ( §3.3), (iv) filtering ( §3.4). note that our method handles su p and ref instances differently, as the large difference in generation space between these two types of instances. ## experiments datasets we generate counterfactual data for hover foot_1 training set @xcite , a multihop dataset with facts sourced from wikipedia. we evaluate the model generalization on three types of development sets, (i) in-domain setting (sourced from wikipedia), including fever @xcite and feverous @xcite . (ii) out-of-domain setting (sourced from specific domains), including politihop (political news) (ostrowski et al., 2021), scifact (scientific articles) @xcite , healthver @xcite and pubhealth (public health) @xcite . (iii) challenge setting (contrastive data), including fm2 @xcite and vitaminc @xcite . details and statistics of datasets are presented in appendix c. baselines we use three types of baselines to augment the hover training set, (i) data augmentation method: eda @xcite . (ii) counterfactual data augmentation methods: crossaug @xcite and polyjuice @xcite . (iii) llms: gpt-3 (text-davinci-003) @xcite and chatgpt (gpt-3.5-turbo-0301) @xcite . more details are presented in appendix d. ## conclusion we present a novel rationale-sensitive pipeline counterfactual data augmentation method (race) to generate logical, diverse, and label-flipping counterfactuals for multi-hop fact verification task. an explain-edit-generate architecture is constructed to generate diverse and logical counterfactual claims based on the rationales. then, a filter process with two modules is employed to further regularize semantic and topic consistency. experimental results reveal the improvement in ood generalization and robustness of the proposed method. intrinsic evaluation and qualitative evaluation of counterfactual claims show that race can generate linguistically diverse and label-flipping counterfactual data while preserving logical relationships.