H o V er: A Dataset for Many-Hop Fact Extraction And Claim Verification (2020)

Abstract: We introduce HoVer (HOppy VERification), a dataset for many-hop evidence extraction and fact verification. It challenges models to extract facts from several Wikipedia articles that are relevant to a claim and classify whether the claim is supported or not-supported by the facts. In HoVer, the claims require evidence to be extracted from as many as four English Wikipedia articles and embody reasoning graphs of diverse shapes. Moreover, most of the 3/4-hop claims are written in multiple sentences, which adds to the complexity of understanding long-range dependency relations such as coreference. We show that the performance of an existing state-of-the-art semantic-matching model degrades significantly on our dataset as the number of reasoning hops increases, hence demonstrating the necessity of many-hop reasoning to achieve strong results. We hope that the introduction of this challenging dataset and the accompanying evaluation task will encourage research in many-hop fact retrieval and information verification.

## introduction the proliferation of social media platforms and digital content has been accompanied by a rise in deliberate disinformation and hoaxes, leading to polarized opinions among masses. with the increasing number of inexact statements, there is a large interest in a fact-checking system that can verify claims based on automatically retrieved facts and evidence. fever @xcite ) is an open-domain fact extraction and verification dataset closely related to this real-world application. however, more than 87% of the claims in fever require information from a single wikipedia article, while real-world "claims" might refer to information from multiple sources. qa datasets like hot-potqa @xcite and qangaroo @xcite represent the first efforts to challenge models to reason with information from three documents at most. however, @xcite and @xcite show that single-hop models can achieve good results in these multi-hop datasets. moreover, most models were also shown to degrade in adversarial evaluation @xcite , where word-matching reasoning shortcuts are suppressed by extra adversarial documents @xcite . in the hotpotqa open-domain setting, the two supporting documents can be accurately retrieved by a neural model exploiting a single hyperlink @xcite . hence, while providing very useful starting points for the community, fever is mostly restricted to a single-hop setting and existing multihop qa datasets are limited by the number of reasoning steps and the word overlapping between the question and all evidence. an ideal multi-hop example should have at least one piece of evidence (supporting document) that cannot be retrieved with high precision by shallowly performing direct semantic matching with only the claim. instead, uncovering this document requires information from previously retrieved documents. in this paper, we try to address these issues by creating hover (i.e., hoppy verification) whose claims (1) require evidence from as many as four english wikipedia articles and (2) contain significantly less semantic overlap between the claims and some supporting documents to avoid reasoning shortcuts.we create hover with 26k claims in three stages. in stage 1 (left box in fig. 1 ), we ask a group of trained and evaluated crowd-workers to rewrite the questionanswer pairs from hotpotqa @xcite into claims that mention facts from two english wikipedia articles. we then introduce extra hops foot_0 to a subset of these 2-hop claims by asking crowdworkers to substitute an entity in the claim with information from another english wikipedia article that describes the original entity. we then repeat this process on these 3-hop claims to further create 4-hop claims. to make many-hop claims more natural and readable, we encourage crowdworkers to write the 3/4-hop claims in multiple sentences and connect them using coreference. an entire evolution history from 2-hop claims to 3/4hop claims is presented in the leftmost box in fig. 1 and in stage 2 (the central box in fig. 1 ), we create claims that are not supported by the evidence by mutating the claims collected in stage 1 with a combination of automatic word/entity substitution and human editing. specifically, we ask the trained crowd-workers to rewrite a claim by making it either more specific/general than or negating the original claim. we ensure the quality of the machine-generated claims using human validation detailed in sec. 2.2. in stage 3, we follow @xcite to label the claims as supported, refuted, or notenoughinfo. however, we find that the decision between re-futed and notenoughinfo can be ambiguous in many-hop claims and even the high-quality, trained annotators from appen, instead of mturk, cannot consistently choose the correct label from these two classes. recent works @xcite have raised concern over the uncertainty of nli tasks with categorical labels and proposed to shift to a probabilistic scale. since this work is mainly targeting the many-hop retrieval, we combine the refuted and notenoughinfo into a single class, namely not-supported. this binary classification task is still challenging for models given the incomplete evidence retrieved, as we will explain later. next, we introduce the baseline system and demonstrate its limited ability in addressing manyhop claims. following a state-of-the-art system @xcite for fever, we build the baseline with a tf-idf document retrieval stage and three bert models fine-tuned to conduct document retrieval, sentence selection, and claim verification respectively. we show that the bi-gram tf-idf @xcite 's top-100 retrieved documents can only recover all supporting documents in 80% of 2-hop claims, 39% of 3-hop claims, and 15% of 4-hop claims. the performance of down-figure 1 : data collection flow chart for hover. in the first stage, we create claims from hotpotqa, validate them and extend to more hops. in the second stage, we apply a variety of mutations to the claims performed by crowd-workers and automatic methods. in the final stage, we ask crowd-workers to label the resulting claims. overall, we provide the community with a novel, challenging and large many-hop fact extraction and claim verification dataset with over 26k claims that can be comprised of multiple sentences connected by coreference, and require evidence from as many as four wikipedia articles. we verify that the claims are challenging, especially in the 3/4hop cases, by showing the limited performance of a state-of-the-art system for both retrieval and verification. we hope that the introduction of hover and the accompanying evaluation task will encourage research in complex many-hop reasoning for fact extraction and claim verification. ## data collection the many-hop fact verification dataset, hover, is a collection of human-written claims about facts in english wikipedia articles created in three main stages (shown in fig. 1 ). in the claim creation stage (sec. 2.1), we ask trained annotators on appen foot_1 to create claims by rewriting question-answer pairs (sec. 2.1.1) from the hotpotqa dataset foot_2 @xcite . the validated 2-hop claims are then extended to (sec. 2.1.2) include facts from more wikipedia articles. in the claim mutation stage (sec. 2.2), claims generated from the above two processes are mutated with human editing and automatic word substitution. finally, in the claim labeling stage (sec. 2.3), trained crowdworkers classify the original and mutated claims as either supported, refuted or notenough-info. we merge the latter two labels into a single not-supported class, owing to ambiguity explained in sec. 2.3. the guidelines and design for every task are shown in the appendix. ## dataset analysis dataset statistics. we partitioned the annotated claims and evidence into training, development, and test sets. the detailed statistics are shown in diverse many-hop reasoning graphs. as questions from hotpotqa @xcite require two supporting documents, our 2-hop claims created from hotpotqa question-answer pairs inherit the same 2-node reasoning graph as shown in the first row in ## baseline system following a state-of-the-art system @xcite on fever @xcite , we build a pipeline system of fact extraction and claim verification. 7 this provides an initial baseline for future works and its performance indicates the many-hop challenge posed by hover. rule-based document retrieval. we use the document retrieval component from @xcite that returns the k closest wikipedia documents for a query using cosine similarity between binned uni-gram and bi-gram tf-idf vectors. this step outputs a set p r of k r document that are processed by downstream neural models. neural-based document retrieval. similar to the retrieval model in @xcite , the bertbase model @xcite takes a single document @xmath0, and outputs a score that reflects the relatedness between p and c. we select a set p n of top k p documents having relatedness scores higher than a threshold of κ p . ## experiments and results we explain the evaluation metrics we use and report the results of the baseline in three evaluation tasks. ## conclusion we present hover, a fact extraction and verification dataset requiring evidence retrieval from as many as four wikipedia articles that form reasoning graphs of diverse shapes. we show that the performance of existing state-of-the-art models degrades significantly on our dataset as the number of reasoning hops increases, hence demonstrating the necessity of robust many-hop reasoning in achieving strong results. we hope that hover will encourage the development of models capable of performing complex many-hop reasoning in the tasks of information retrieval and verification. document retrieval, sentence selection, and claim verification. the fine-tuning is done with a batch size of 16 and the default learning rate of 5e-5 without warmup. we set k @xmath1, k @xmath2, κ @xmath3.5, and κ @xmath4.3 based on the memory limit and the dev set performance. we select our system with the best dev-set verification accuracy and report its scores on the hidden test set. the entire pipeline is visualized in fig. 2 . for document retrieval and sentence selection tasks, we fine-tune the bert on 4 nvidia v100 gpus for 3 epochs. the training of both tasks takes around 1 hour. for claim verification task, we fine-tune the bert on a single nvidia v100 for 3 epochs. the training finishes in 30 minutes.