Reasoning Over Semantic-Level Graph for Fact Checking (2020)

Abstract: Fact checking is a challenging task because verifying the truthfulness of a claim requires reasoning about multiple retrievable evidence. In this work, we present a method suitable for reasoning about the semantic-level structure of evidence. Unlike most previous works, which typically represent evidence sentences with either string concatenation or fusing the features of isolated evidence sentences, our approach operates on rich semantic structures of evidence obtained by semantic role labeling. We propose two mechanisms to exploit the structure of evidence while leveraging the advances of pre-trained models like BERT, GPT or XLNet. Specifically, using XLNet as the backbone, we first utilize the graph structure to re-define the relative distances of words, with the intuition that semantically related words should have short distances. Then, we adopt graph convolutional network and graph attention network to propagate and aggregate information from neighboring nodes on the graph. We evaluate our system on FEVER, a benchmark dataset for fact checking, and find that rich structural information is helpful and both our graph-based mechanisms improve the accuracy. Our model is the state-of-the-art system in terms of both official evaluation metrics, namely claim verification accuracy and FEVER score.

## introduction internet provides an efficient way for individuals and organizations to quickly spread information to massive audiences. however, malicious people spread false news, which may have significant influence on public opinions, stock prices, even presidential elections @xcite . @xcite show that false news reaches more people than the truth. the situation is more urgent as advanced pre-trained language models @xcite can produce remarkably coherent and fluent texts, which lowers the barrier for the abuse of creating deceptive content. in this paper, we study fact checking with the goal of automatically assessing the truthfulness of a textual claim by looking for textual evidence. in the usa" from the second evidence. it is therefore desirable to mine the semantic structure of evidence and leverage it to verify the truthfulness of the claim. under the aforementioned consideration, we present a graph-based reasoning approach for fact checking. with a given claim, we represent the retrieved evidence sentences as a graph, and then use the graph structure to guide the reasoning process. specifically, we apply semantic role labeling (srl) to parse each evidence sentence, and establish links between arguments to construct the graph. when developing the reasoning approach, we intend to simultaneously leverage rich semantic structures of evidence embodied in the graph and powerful contextual semantics learnt in pre-trained models like bert @xcite , gpt @xcite and xlnet @xcite . to achieve this, we first re-define the distance between words based on the graph structure when producing contextual representations of words. furthermore, we adopt graph convolutional network and graph attention network to propagate and aggregate information over the graph structure. in this way, the reasoning process employs semantic representations at both word/sub-word level and graph level. we conduct experiments on fever @xcite , which is one of the most influential benchmark datasets for fact checking. fever consists of 185,445 verified claims, and evidence sentences for each claim are natural language sentences from wikipedia. we follow the official evaluation protocol of fever, and demonstrate that our approach achieves state-of-the-art performance in terms of both claim classification accuracy and fever score. ablation study shows that the integration of graph-driven representation learning mechanisms improves the performance. we briefly summarize our contributions as follows. ## task definition and pipeline with a textual claim given as the input, the problem of fact checking is to find supporting evidence sentences to verify the truthfulness of the claim. we conduct our research on fever @xcite , short for fact extraction and verification, a benchmark dataset for fact checking. systems are required to retrieve evidence sentences from wikipedia, and predict the claim as "sup-ported ", "refuted " or "not enough info (nei) ", standing for that the claim is supported by the evidence, refuted by the evidence, and is not verifiable, respectively. there are two official evaluation metrics in fever. the first is the accuracy for three-way classification. the second is fever score, which further measures the percentage of correct retrieved evidence for "sup-ported " and "refuted " categories. both the statistic of fever dataset and the equation for calculating fever score are given in appendix b. here, we present an overview of our pipeline for fever, which follows the majority of previous studies. our pipeline consists of three main components: a document retrieval model, a sentence-level evidence selection model, and a claim verification model. figure 2 gives an overview of the pipeline. with a given claim, the document retrieval model retrieves the most related documents from a given collection of wikipedia documents. with retrieved documents, the evidence selection model selects top-k related sentences as the evidence. finally, the claim verification model takes the claim and evidence sentences as the input and outputs the veracity of the claim. ## graph-based reasoning approach in this section, we introduce our graph-based reasoning approach for claim verification, which is the main contribution of this paper. taking a claim and retrieved evidence sentences foot_0 as the input, our approach predicts the truthfulness of the claim. for fever, it is a three-way classification problem, which predicts the claim as "supported ", "re-futed " or "not enough info (nei) ". the basic idea of our approach is to employ the intrinsic structure of evidence to assess the truthfulness of the claim. as shown in the motivating example in figure 1 , making the correct prediction needs good understanding of the semantic-level structure of evidence and the reasoning process based on that structure. in this section, we first describe our graph construction module ( §3.1). then, we present how to apply graph structure for fact checking, including a contextual representation learning mechanism with graph-based distance calculation ( §3.2), and graph convolutional network and graph attention network to propagate and aggregate information over the graph ( §3.3 and §3.4). ## document retrieval and evidence selection in this section, we briefly describe our document retrieval and evidence selection components to make the paper self contained. ## experiments we evaluate on fever @xcite , a benchmark dataset for fact extraction and verification. each instance in fever dataset consists of a claim, groups of ground-truth evidence from wikipedia and a label (i.e., "sup-ported ", "refuted " or "not enough info (nei) "), indicating its veracity. fever includes a dump of wikipedia, which contains 5,416,537 pre-processed documents. the two official evaluation metrics of fever are label accuracy and fever score, as described in section 2. label accuracy is the primary evaluation metric we apply for our experiments because it directly measures the performance of the claim verification model. we also report fever score for comparison, which measures whether both the predicted label and the retrieved evidence are correct. no evidence is required if the predicted label is nei. ## related work in general, fact checking involves assessing the truthfulness of a claim. in literature, a claim can be the majority of participating teams in the fever challenge @xcite use the same pipeline consisting of three components, namely document selection, evidence sentence selection, and claim verification. in document selec-tion phase, participants typically extract named entities from a claim as the query and use wikipedia search api. in the evidence selection phase, participants measure the similarity between the claim and an evidence sentence candidate by training a classification model like enhanced lstm @xcite in a supervised setting or using string similarity function like tfidf without trainable parameters. @xcite utilizes semantic frames for evidence selection. in this work, our focus is the claim classification phase. top-ranked three systems aggregate pieces of evidence through concatenating evidence sentences into a single string @xcite , classifying each evidence-claim pair separately, merging the results @xcite , and encoding each evidence-claim pair followed by pooling operation @xcite . @xcite are the first to use bert to calculate claim-specific evidence sentence representations, and then develop a graph network to aggregate the information on top of bert, regarding each evidence as a node in the graph. our work differs from @xcite in that (1) the construction of our graph requires understanding the syntax of each sentence, which could be viewed as a more fine-grained graph, and (2) both the contextual representation learning module and the reasoning module have model innovations of taking the graph information into consideration. instead of training each component separately, @xcite show that joint learning could improve both claim verification and evidence selection. ## conclusion in this work, we present a graph-based approach for fact checking. when assessing the veracity of a claim giving multiple evidence sentences, our approach is built upon an automatically constructed graph, which is derived based on semantic role labeling. to better exploit the graph information, we propose two graph-based modules, one for calculating contextual word embeddings using graph-based distance in xlnet, and the other for learning representations of graph components and reasoning over the graph. experiments show that both graph-based modules bring improvements and our final system is the state-of-the-art on the public leaderboard by the time our paper is submitted. evidence selection is an important component of fact checking as finding irrelevant evidence may lead to different predictions. a potential solution is to jointly learn evidence selection and claim verification model, which we leave as a future work.