Generating Fact Checking Summaries for Web Claims (2020)

Abstract: We present SUMO, a neural attention-based approach that learns to establish correctness of textual claims based on evidence in the form of text documents (e.g., news articles or web documents). SUMO further generates an extractive summary by presenting a diversified set of sentences from the documents that explain its decision on the correctness of the textual claim. Prior approaches to address the problem of fact checking and evidence extraction have relied on simple concatenation of claim and document word embeddings as an input to claim driven attention weight computation. This is done so as to extract salient words and sentences from the documents that help establish the correctness of the claim. However this design of claim-driven attention fails to capture the contextual information in documents properly. We improve on the prior art by using improved claim and title guided hierarchical attention to model effective contextual cues. We show the efficacy of our approach on political, healthcare, and environmental datasets.

## introduction most of the information consumed by the world is in the form of digital news, blogs, and social media posts available on the web. however, most of this information is written in the absence of facts and evidences. our ever-increasing reliance on information from the web is becoming a severe problem as we base our personal decisions relating to politics, environment, and health on unverified information available online. for example, consider the following unverified claim on the web: "smoking may protect against covid-19." a user attempting to verify the correctness of the above claim will often take the following steps: prior approaches to automatic fact checking rely on predicting the credibility of facts @xcite , instance detection @xcite , and fact entailment in supporting documents @xcite . the majority of these methods rely on linguistic features @xcite @xcite , social contexts, or user responses @xcite and comments. however, these approaches do not help explain the decisions generated by the machine learning models. recent works such as @xcite @xcite overcome the explainability gap by extracting snippets from text documents that support or refute the claim. @xcite apply claim-based and latent aspect-based attention to model the context of text documents. @xcite model latent aspects such as the speaker or author of the claim, topic of the claim, and domains of retrieved web documents for the claim. we observe in our experiments that in prior works @xcite , the design of claim guided attention in these methods is not effective and latent aspects such as the topic and speaker of claims are not always available. the snippets extracted by such models are not comprehensive or topically diverse. to overcome these limitations, we propose a novel design of claim and document the current evidence suggests that the severity of covid is higher among smokers, prevent the health risk linked to the excessive consumption or misuse" of nicotine products by people hoping to protect themselves from covid-19. evidence from china, where covid-19 originated, shows that people who have cardiovascular and respiratory conditions caused by tobacco use, or otherwise, are at higher risk of developing severe covid-19 symptoms. ho urges researchers, scientists and the media to be cautious about amplifying unproven claims that tobacco or nicotine could reduce the risk of covid-19. smoking is also associated with increased development of acute respiratory distress syndrome, a key complication for severe cases of covid-19. ## related work we now describe prior work related to our problem setting. first, we describe works that rely only on features derived from documents that support the input textual claim. second, we describe works that additionally include features derived from social media posts in connection to the claim. third and finally, we describe works that rely on extracting textual snippets from text documents to explain a model's decision on the claim's correctness. ## sumo we now formally describe the task of fact checking and explain sumo in detail. sumo works in two stages. in the first stage, it predicts the correctness of the claim. in the second stage, it generates a topically diverse summary for the claims. as input, we are provided with a web claim @xmath0, where c is a collection of web claims and a pseudo-relevant set of documents @xmath1, d 2 , . . . , d m }, where m is the number of results retrieved for claim c. the documents @xmath2, using claim c as a query. each retrieved document d is accompanied by its title t and text body bd, i.e. (@xmath3, bd ). we define the representation of each document's body as a collection of k sentences as @xmath4, s 2 , ..., s k } and each sentence as the collection of l words as {w 1 , w 2 , ..., w l } ∈ w, where w is the overall word vocabulary of the corpus. by k and l, we denote the maximum numbers of sentences in a document and the maximum number of words in a sentence, respectively. we use both word2vec and pre-trained glove embeddings to obtain the vector representations for each claim, title, and document body. the objective is to classify the claim as either true or false and automatically generate a topically diverse summary pieced together from d for establishing the correctness of the claim. ## evaluation datasets. we use two publicly available datasets, namely politifact political claims dataset and snopes political claims dataset @xcite for evaluating sumo's capability for fact checking. dataset statistics for both the datasets are shown in input • the ozone layer is healing. figure 3 : examples from climate change and health care dataset care that additionally contains claims pertaining to the current global covid-19 pandemic from healthfeedback.org. examples of claims from these two datasets are shown in figure 3 . we make the new datasets, publicly available to the research community at the following url: https: //github.com/rahulomishra/sumo/ . ## conclusion we presented sumo, a neural network based approach to generate explainable and topically diverse summaries for verifying web claims. sumo uses an improved version of hierarchical claimdriven attention along with title-driven and selfattention to learn an effective representation of the external evidences retrieved from the web. learning this effective representation in turn assists us in establishing the correctness of textual claims. using the overall attention weights from the novel atop attention method and topical distributions of the sentences, we generate extractive summaries for the claims. in addition to this, we release two important datasets pertaining to climate change and healthcare claims. in future, we plan to investigate the bert @xcite and other transformer @xcite architecture based embedding meth-ods in place of glove @xcite embeddings for better contextual representation of words.