Zero-shot Fact Verification by Claim Generation (2021)

Abstract: Neural models for automated fact verification have achieved promising results thanks to the availability of large, human-annotated datasets. However, for each new domain that requires fact verification, creating a dataset by manually writing claims and linking them to their supporting evidence is expensive. We develop QACG, a framework for training a robust fact verification model by using automatically generated claims that can be supported, refuted, or unverifiable from evidence from Wikipedia. QACG generates question-answer pairs from the evidence and then converts them into different types of claims. Experiments on the FEVER dataset show that our QACG framework significantly reduces the demand for human-annotated training data. In a zero-shot scenario, QACG improves a RoBERTa model’s F1 from 50% to 77%, equivalent in performance to 2K+ manually-curated examples. Our QACG code is publicly available.

## introduction fact verification aims to validate a claim in the context of evidence. this task has attracted growing interest with the rise in disinformation in news and social media. rapid progress has been made by training large neural models @xcite @xcite on the fever dataset @xcite , containing more than 100k human-crafted (evidence, claim) pairs based on wikipedia. fact verification is demanded in many domains, including news articles, social media, and scientific documents. however, it is not realistic to assume that large-scale training data is available for every new domain that requires fact verification. creating training data by asking humans to write claims and search for evidence to support/refute them can be extremely costly. we address this problem by exploring the possibility of automatically generating large-scale (evidence, claim) pairs to train the fact verification model. we propose a simple yet general framework question answering for claim generation (qacg) to generate three types of claims from any given evidence: 1) claims that are supported by the evidence, 2) claims that are refuted by the evidence, and 3) claims that the evidence does not have enough information (nei) to verify. to generate claims, we utilize question generation (qg) @xcite @xcite , which aims to automatically ask questions from textual inputs. qg has been shown to benefit various nlp tasks, such as enriching qa corpora @xcite , checking factual consistency for summarization @xcite , and data augmentation for semantic parsing @xcite . to the best of our knowledge, we are the first to employ qg for fact verification. as illustrated in figure 1 , given a passage p as the evidence, we first employ a question generator to generate a question-answer pair (q, a) for the evidence. we then convert (q, a) into a claim c (qa-to-claim) based on the following logical assumptions: a) if p can answer q and a is the correct answer, then c is a supported claim; b) if p can answer q but a is an incorrect answer, then c is a refuted claim; c) if p cannot answer q, then c is a nei claim. the question generator and the qa-to-claim model are off-the-shelf bart models @xcite , finetuned on squad @xcite and qa2d @xcite datasets. we generate 100k (evidence, claim) pairs for each type of claim, which we then use to train a roberta @xcite model for fact verification. we evaluate the model on three test sets the 1992 los angeles riots, also known as the rodney king riots were a series of riots, lootings, arsons, and civil disturbances that occurred in los angeles county, california in april and may 1992. by the time the riots ended, 63 people had been killed. based on the fever dataset. although we do not use any human-labeled training examples, the model achieves over 70% of the f 1 performance of a fully-supervised setting. by finetuning the model with only 100 labeled examples, we further close the performance gap, achieving 89.1% of fullysupervised performance. the above results show that pretraining the fact verification model with generated claims greatly reduces the demand for in-domain human annotation. when evaluating the model on an unbiased test set for fever, we find that training with generated claims also produces a more robust fact verification model. ## methodology given a claim c and a piece of evidence p as inputs, a fact verification model f predicts a label @xmath0, refuted, nei} to verify whether c is supported, refuted, or can not be verified by the information in p. for the zero-shot setting, we assume no humanannotated training example is available. instead, we generate a synthetic training set based on our qacg framework to train the model. ## experiments by applying our qacg model to each of the 18, 541 wikipedia articles in the fever training set, we generate a total number of 176, 370 supported claims, 360, 924 refuted claims, and 258, 452 nei claims. our generated data is around five times the size of the human-annotated claims in fever. we name this generated dataset as qacg-full. we then create a balanced dataset qacg-filtered by randomly sampling 100, 000 samples for each class. statistics of the fever and the generated dataset are in appendix b. evaluation datasets. we evaluate fact verification on three different test sets based on fever: 1) fever-s/r: since only the supported and refuted claims are labeled with gold evidence in fever, we take the claim-evidence pairs of these two classes from the fever test set for evaluation. 2) fever-symmetric: this is a carefullydesigned unbiased test set designed by @xcite to detect the robustness of the fact verification model. note that only supported and refuted claims are present in this test set. 3) fever-s/r/n: the full fever test set are used for a three-class verification. we follow @xcite to use the system of @xcite to retrieve evidence sentences for nei claims. fact verification models. as shown in we employ four baselines that also do not need any human-annotated claims to compare with our method. random guess (u1) is a weak baseline that randomly predicts the class label. gpt2 perplexity (u2) predicts the class label based on the perplexity of the claim under a pretrained gpt2 @xcite language model, following the assumption that "misinformation has high perplexity" @xcite . mnli-transfer (u3) trains a bert model for natural language inference on the multinli corpus @xcite and applies it for fact verification. lm as fact checker @xcite ) (u4) leverages the implicit knowledge stored in the pretrained bert language model to verify a claim. the implementation details are given in appendix c. ## conclusion and future work we utilize the question generation model to ask different questions for given evidence and convert question-answer pairs into claims with different labels. we show that the generated claims can train a well-performing fact verification model in both the zero-shot and the few-shot learning setting. potential future directions could be: 1) generating more complex claims that require deep reasoning; 2) extending our framework to other fact checking domains beyond wikipedia, e.g., news, social media; 3) leveraging generated claims to improve the robustness of fact checking systems. take a as its correct answer. this is the premise of generating a correct supported claim. therefore, we specially evaluate this answerability property via human ratings. we randomly sample 100 generated question-answer pairs with their corresponding evidence and ask two workers to judge the answerability of each sample. we do this for both the nqg++ model and the bart model. to investigate the impact of question quality on the fact verification performance, we separately use the nqg++ and bart as the question generator to generate claims and train the roberta model. the performance is summarized in