H ierarchical E vidence S et M odeling for Automated Fact Extraction and Verification (2020)

Abstract: Automated fact extraction and verification is a challenging task that involves finding relevant evidence sentences from a reliable corpus to verify the truthfulness of a claim. Existing models either (i) concatenate all the evidence sentences, leading to the inclusion of redundant and noisy information; or (ii) process each claim-evidence sentence pair separately and aggregate all of them later, missing the early combination of related sentences for more accurate claim verification. Unlike the prior works, in this paper, we propose Hierarchical Evidence Set Modeling (HESM), a framework to extract evidence sets (each of which may contain multiple evidence sentences), and verify a claim to be supported, refuted or not enough info, by encoding and attending the claim and evidence sets at different levels of hierarchy. Our experimental results show that HESM outperforms 7 state-of-the-art methods for fact extraction and claim verification. Our source code is available at https://github.com/ShyamSubramanian/HESM .

## introduction a study by @xcite has revealed that 60% of people on social media share the news after reading just the title, without reading the actual content of the news. unfortunately, the rise of social media has further accelerated the communication and propagation of unverified information. to solve the problem, our work focuses on automated fact extraction and verification task, which requires retrieving the evidence related to a claim as well as verifying the claim based on the evidence. the task is challenging since it requires semantic understanding and reasoning to learn the subtleties that differ between evidence that supports and evidence that refutes a claim. the task's difficulty is further amplified for claims that require aggregating information from multiple evidence sentences in different documents. previous works, in fact verification, either operate by combining all the evidence sentences @xcite or they operate at each evidence sentence-level and aggregate them later @xcite . combining all the sentences together may lead to the combination of redundant, noisy, and irrelevant information with the relevant information. this makes claim verification more complicated in terms of identifying and learning the context of only the relevant sentences. on the other hand, processing each evidence sentence separately, delays the combination of relevant sentences that belong to the same evidence set, for claims that require aggregating information from multiple sentences. it also makes claim verification harder since it summarizes information without complete context. figure 1 depicts an example of an ideal verification system, which extracts evidence sets, processes them individually, and then aggregates them later. in the example, four evidence sentences are retrieved. sentences which are relevant and hyperlinked, are combined to form evidence sets (called evidence set @xcite and evidence set @xcite in the figure). each evidence set verifies the claim individually, and then they are aggregated for the final verification. like figure 1 , our proposed framework also retrieves and combines evidence sentences into evidence sets in an iterative fashion. then, it processes each evidence set individually to form a representation of the evidence set using word-level attention. then, it combines information from all the evidence set representations using contextual and non-contextual aggregation methods, which use evidence set-level attention. the word-level attention, along with evidence set-level attention, forms a hierarchical attention mechanism. finally, our figure 1 : an example of claim, evidence sets, and verdict. the arrows represent the hierarchy of the fact extraction and verification process. the second sentence in each evidence set is retrieved from a document hyperlinked from the first sentence. framework learns to verify the claim at different levels of hierarchy (i.e., at each evidence set-level and the aggregated evidence level). our main contributions are as follows: several works exist in fact verification based on different forms of claim and evidence. thorne and vlachos (2017); vlachos and riedel (2015) verify numerical claims using subject-predicateobject triples from knowledge graph as evidence. nakashole and mitchell (2014); bast et al. (2017) verify subject-predicate-object triple based claims. chen et al. (2020) verifies textual claims based on evidences in a tabular format. fact verification is studied in different natural language settings namely recognizing textual entailment @xcite , natural language inference @xcite and claim verification @xcite . a differently motivated but closely related problem is fact checking in journalism, also known as fake news detection @xcite . in this work, we focus on claim verification using the fever dataset @xcite with textual claims and evidence. previous works on the fact extraction and claim verification task follow a three-stage pipeline that includes document retrieval, evidence sentence retrieval and claim verification. most previous works reuse the document retrieval component of topperforming systems @xcite @xcite in the fever shared task 1.0 challenge @xcite . evidence sentence retrieval component in almost all previous works retrieves all the evidences through a single iteration @xcite @xcite @xcite . @xcite uses a multi-hop retrieval strategy through two iterations to retrieve evidence sentences that are conditioned on the retrieval of other evidence sentences. then, they choose all the top-most relevant evidence sentences with the highest relevance scores and combine them. our work follows a similar strategy, but differs from the prior work by combining only evidence sentences that belong to the same evidence set, and then processing each evidence set separately. in claim verification component, @xcite al. (2019); yoneda et al. (2018); hanselowski et al. (2018b) use a modified esim model @xcite for verification. recent works @xcite @xcite use bert model @xcite for claim verification. few other works @xcite use graph based models for fine-grained semantic reasoning. different from the previous works, our model operates with claimevidence set pairs instead of claim-evidence sentence pairs. our model benefits from encoding, attending and evaluating at different levels of hi-erarchy, as well as from both contextual and noncontextual aggregations of the evidence sets. ## problem definition given a set of m textual documents and a claim c i , the problem is to find a set of evidence sentences êi = {s 1 , s 2 , ..., s | êi | } and classify the claim c i as ŷi ∈ {s, r, n ei} (i.e., supported, re-futed or not enough info). for a successful verification of the claim c i , there are two conditions: (1) êi should match at least one evidence set in the ground truth evidence sets e i and (2) ŷi should match the ground truth entailment label y i . ## hierarchical evidence set modeling our hierarchical evidence set modeling (hesm) framework consists of three components namely document retriever, multi-hop evidence retriever and claim verification. figure 2 shows an overview of our framework. the document retriever component retrieves the top k 1 documents that are relevant to the claim. the multi-hop retriever component retrieves the relevant top k 2 evidence sets from the k 1 retrieved documents via an iterative fashion. the claim verification component classifies the claim as supports, refutes or not enough info based on the retrieved evidence sets. following prior works, in our framework, we reuse the document retriever component from @xcite , which works well in terms of relevant document retrieval. we mainly focus on and propose novel multi-hop evidence retriever and claim verification components.