EX - FEVER : A Dataset for Multi-hop Explainable Fact Verification (2024)

Abstract: Fact verification aims to automatically probe the veracity of a claim based on several pieces of evidence. Existing works are always engaging in accuracy improvement, let alone explainability, a critical capability of fact verification systems.Constructing an explainable fact verification system in a complex multi-hop scenario is consistently impeded by the absence of a relevant, high-quality dataset. Previous datasets either suffer from excessive simplification or fail to incorporate essential considerations for explainability. To address this, we present EX-FEVER, a pioneering dataset for multi-hop explainable fact verification. With over 60,000 claims involving 2-hop and 3-hop reasoning, each is created by summarizing and modifying information from hyperlinked Wikipedia documents. Each instance is accompanied by a veracity label and an explanation that outlines the reasoning path supporting the veracity classification. Additionally, we demonstrate a novel baseline system on our EX-FEVER dataset, showcasing document retrieval, explanation generation, and claim verification, and validate the significance of our dataset. Furthermore, we highlight the potential of utilizing Large Language Models in the fact verification task. We hope our dataset could make a significant contribution by providing ample opportunities to explore the integration of natural language explanations in the domain of fact verification.

## introduction fact verification, also known as fact checking, is a task to predict the veracity of a claim based on retrieved evidence, i.e., evidence that supports the claim, refutes the claim, or has insufficient information to judge the claim. since the misinformation is widely spread with the proliferation of social platforms, recent years have witnessed the rapid development of automatic fact checking over various domains, such as politics @xcite @xcite , public health @xcite @xcite , and science @xcite . a typical fact-checking system consists of two main stages: evidence retrieval and veracity prediction. the evidence retrieval stage aims to improve the recall of golden evidence. the veracity prediction is made based on the interaction between the given claim and the retrieved evidence. the first large-scale fact-checking dataset, fever @xcite , has significantly contributed to the promotion of existing works in the field where claims are annotated by crowd workers and rely on information sourced from wikipedia articles. the fact-checking systems inspired by fever aim to enhance both the performance (e.g., precision and recall) of evidence retrieval and the accuracy of verdict prediction @xcite . however, it is important to acknowledge that over 87% of the claims in fever rely on information sourced from a single wikipedia article. in contrast, real-world claims often involve information from multiple sources, making it challenging for single-hop models to reason effectively without resorting to word-matching shortcuts @xcite . to address this limitation, the hover dataset is introduced by @xcite . the hover dataset is derived from the qa dataset @xcite , where claims require evidence from up to four english wikipedia articles. the multi-hop design presents numerous challenges for both retrieval and verification models, but its oversimplification of the verification task as a binary classification is a major reason for its limited adoption in the fact verification task. though the research on multi-hop complex reasoning, the explainability is still under-explored. explainability plays a significant role in factchecking models for two main reasons. firstly, displaying the veracity prediction along with the corresponding textual explanations can make the fact-checking system more credible to human users @xcite . moreover, the performance (e.g., f1 score, accuracy) of models can be improved when the explanation is fed @xcite . hence, shedding light on the research of explainability in fact checking holds great value. ## related work in this section, we briefly review representative works in the field of fact verification and model explainability. ## data collection in this section, we provide a detailed description of the specific steps involved in annotating claims, including their corresponding labels and explanations. ## baseline system description our baseline system comprises three stages. firstly, given a claim, the system aims to provide the most relevant documents as evidence. secondly, the system summarizes the information from multiple documents into a concise summary, which then serves as the system's output explanation. finally, based on the interaction between the claim and the summary, the system generates a verdict. in the subsequent sections, we will delve into the specific claim: john mayer is an american singer-songwriter whose debut ep was later re-released by an american record label owned by sony music entertainment. ## prompt based approach with the plms (pre-trained language models) and gpt model series development @xcite @xcite , large language models (llms) exhibit immense potential in many general tasks, especially with suitable prompts @xcite . in this section, we do some preliminary investigations exploring using llms in the fact checking task in two directions: directly using llms as an actor, and using llms as a planner also we both evaluate the verdict accuracy and the ability of llms to generate explanations. ## conclusion we introduce a publicly accessible, extensive factchecking dataset, named ex-fever, encompassing over 60,000 intricate multi-hop claims. for each claim's veracity, we provide an elucidating annotation to facilitate human comprehension of the adjudication process. we devise a comprehensive system using ex-fever, encompassing a retrieval phase, a summarization component serving as the explanatory stage, and a subsequent verification stage. the experiment results validate the challenge and the importance of ex-fever. furthermore, we do some preliminary investigations exploring using llms in the fact checking task. we elect the gpt-3.5-turbo model to represent large language models. utilizing two distinct settings using llms as an actor and using llms as a planner, we evaluated the llm on our dataset mini-test set. we find that the llm exhibits better performance when utilized as planners rather than directly em-ployed as actors and that llm excels in generating explanations rather than making predictions. the findings reveal that despite the prowess of llms, there remains significant potential for improvement. in summary, ex-fever could serve as a valuable benchmark in studying the explainable multi-hop fact-checking task by improving reliability, trustworthiness, and facilitating better decision-making across different domains.