C onstrained F act V erification for FEVER (2020)

Abstract: Fact-verification systems are well explored in the NLP literature with growing attention owing to shared tasks like FEVER. Though the task requires reasoning on extracted evidence to verify a claim’s factuality, there is little work on understanding the reasoning process. In this work, we propose a new methodology for fact-verification, specifically FEVER, that enforces a closed-world reliance on extracted evidence. We present an extensive evaluation of state-of-the-art verification models under these constraints.

## introduction a rapid increase in the spread of misinformation on the internet has necessitated automated solutions to determine the validity of a given piece of information. to this end, the fact extraction and verification (fever) shared task @xcite 1 introduced a dataset for evidencebased fact verification. given a claim, the task involves extracting relevant evidence sentences from a given wikipedia dump and assigning a label to the claim by reasoning over the extracted evidence (supports / refutes / notenoughinfo). several recent works @xcite @xcite leverage representations from large pre-trained language models (lms) like bert @xcite , and roberta @xcite to achieve state-of-the-art results on fever. however, it is unclear how factual knowledge encompassed in these lms influences the verification process. more recently, @xcite developed a fact verification system solely based on large pretrained lms and presented their superior zero-shot performance on fever compared to a random baseline. this result clearly shows the influence of factual knowledge embedded inside these lms, but relying entirely on such knowledge directly contrasts to the evidence-based paradigm of factverification. such reliance can be problematic, especially with evolving evidence (wikipedia pages are constantly updated to reflect the latest events). @xcite illustrate this phenomenon through an example fact, "halep failed to ever win a wimbledon title", which was valid until july 2019 but not thereafter. in this work, we propose methods to train factverification models that explicitly reason on the available evidence instead of relying on the factual knowledge in pre-trained lms, thereby emulating a closed-world setting. this is particularly important in the context of the fever dataset because of the overlap between the source corpus used for compiling fever and the ones commonly used to pre-train lms (wikipedia). we build upon the work of clark et al. (2020) that demonstrated the ability of transformers (bert, roberta) to function as soft theorem provers. they induce a closed-world reasoning process by fine-tuning on a carefully curated synthetic natural language rulebase. in this work, we transfer this ability to fever and gauge the feasibility of such closed-world reasoning. additionally, we also construct an entity-anonymized version of fever following @xcite for evaluating our proposed models. we construct the anonymized version by masking prominent named entities in the claim-evidence pairs, thereby reducing any reliance on pre-trained factual knowledge. our experiments adopt the popular three-stage pipeline of fever task, comprising document selection, evidence sentence extraction, and claim verification @xcite . we primarily focus on the claim verification stage of fever, while using the state-of-the-art document selec-tion and evidence sentences extraction from @xcite . our focus is motivated since only the claim verification step involves a joint (often complicated) reasoning over the extracted evidence. our main contributions are, our datasets and code are publicly available. foot_0 ## constrained verification traditionally, most fever systems rely on large pre-trained language models (lms) to encode the claim and extracted evidence sentences. previously, @xcite studied various reasons for the surprisingly good performance of claim-only classifiers on fever and reported dataset idiosyncrasies to be the primary reason as opposed to world knowledge in word embeddings. however, they present only a preliminary analysis of the impact of world knowledge from glove embeddings @xcite . in this work, we present an in-depth analysis because the issue is particularly relevant in the context of large pre-trained lms. to the best of our knowledge, we are not aware of any other works that look into the impact of embedding's world knowledge on fever. in a nutshell, we model the task under a closedworld setting with the extracted evidence as the only available factual information to the model. overall, we believe the methods proposed in this paper are general enough to apply to any factverification task. however, we show a case study only on fever due to its wide-spread popularity. to this end, we first present an entityanonymized version of the fever dataset and then propose pre-training strategies to enforce the above described closed-world setting on fever models. ## related work we adopted the widely used document selection method from @xcite . many recent state-of-the-art fever systems involve reasoning over evidence graphs @xcite @xcite along with competitive lmbased models @xcite . dataset specific idiosyncrasies have been identified in fever @xcite as well as in nli @xcite @xcite , but is not the focus of this work. ## conclusion we identify a critical issue with existing claim verification systems, especially the recent models that utilize large pre-trained lms. we propose to perform fact verification under a closed-world setting and present our results on the task of fever. while it is hard to evaluate the reliance on implicit pretrained knowledge, our initial results indicate that such reliance is helpful for fever.