Harnessing Abstractive Summarization for Fact-Checked Claim Detection (2022)

Abstract: Social media platforms have become new battlegrounds for anti-social elements, with misinformation being the weapon of choice. Fact-checking organizations try to debunk as many claims as possible while staying true to their journalistic processes but cannot cope with its rapid dissemination. We believe that the solution lies in partial automation of the fact-checking life cycle, saving human time for tasks which require high cognition. We propose a new workflow for efficiently detecting previously fact-checked claims that uses abstractive summarization to generate crisp queries. These queries can then be executed on a general-purpose retrieval system associated with a collection of previously fact-checked claims. We curate an abstractive text summarization dataset comprising noisy claims from Twitter and their gold summaries. It is shown that retrieval performance improves 2x by using popular out-of-the-box summarization models and 3x by fine-tuning them on the accompanying dataset compared to verbatim querying. Our approach achieves Recall@5 and MRR of 35% and 0.3, compared to baseline values of 10% and 0.1, respectively. Our dataset, code, and models are available publicly: https://github.com/varadhbhatnagar/FC-Claim-Det/ .

## introduction social media is increasingly used for business, entertainment, and political discourse, thus, encouraging users to produce and consume large volumes of information that may not always be accurate. due to a lack of digital awareness, the masses often believe and forward such disputed claims in their social circles. such spread of misinformation often culminates in incidents which cause damage to life and property. it is well documented that misinformation is used as a tool by political agents to slander their opposition @xcite and influence the opinion of the masses. it becomes furthermore dangerous when such claims pertain to religious beliefs, often leading to violence and mob lynchings 1 . in the era of covid-19, unverified medical advice has also been circulated on social media @xcite which has already led to various health hazards. social media platforms have undertaken concerted efforts to tackle the fake news epidemic by enforcing strict policies to weed out unverified and sensitive content and ban habitual offenders. journalistic organizations such as alt news 2 , factly 3 , boom live 4 and snopes 5 among others are also fighting this problem by publishing fact-checking articles investigating the veracity of viral dubious claims. these articles detail the journalistic procedures followed to fact-check the claim along with suitable references. numerous researchers are working on aibased solutions for fact-checking claims. many datasets @xcite @xcite have been released to train models which can automate subtasks such as claim verification, evidence retrieval and assigning a verdict in a fact-checking workflow. a critical and insufficiently researched step in the fact-checking workflow is-detecting whether a claim has been fact-checked previously. this is a repetitive task with immense scope for automation, shrinking the turnaround time for a claim and ensuring that human efforts are put to better use on tasks involving higher cognition, such as assigning a verdict. in literature, learning-to-rank models @xcite @xcite have been proposed for this step, which on being queried, produce a ranked list of results from a closed dataset of previously verified claims. dozens of fact-checking articles are being published every hour around the world. it is difficult for journalistic organizations to maintain such a large real-time collection of fact-checked articles and claims, thus, making such an approach infeasible in real-world scenarios. we propose a novel workflow (as shown in figure 1 ) for detecting previously fact-checked claims. in this work, we use google's fact check explorer, a cross-publisher, cross-language search engine for previously fact-checked articles, as a black box retriever. as social media platforms contribute a great deal to spreading misinformation, we deal with naturally occurring textual claims on twitter in this study as opposed to artificial, well defined and structured claims @xcite . instead of querying using verbatim claims, which are noisy, it is proposed that abstractive text summarization be used as a precursor to querying to generate clear, succinct queries capturing the claim in a minimum number of words. figure 1 represents a subpart of the complete factchecking pipeline @xcite with our proposed changes in red. ccr and smc are defined in section 3. in literature, no dataset exists for abstractive summarization of tweets, and no attempts have been made to address this problem using the fact check explorer to the best of our knowledge. our contributions can be distilled into the following: a general-purpose four-step automatic factchecking pipeline is presented by @xcite . the task of determining if a claim has been previously fact-checked is the second step in the pipeline. this problem is addressed in a series of open challenges @xcite at checkthat! workshop @xcite @xcite as part of clef foot_1 . @xcite collect, annotate and release datasets of claim pairs and evidence sets, sourced from politifact foot_2 and snopes for solving this task. they develop and demonstrate the robustness of bm25 and bert @xcite based learning to rank models on their dataset for this task. vo and lee (2020); @xcite also propose variants of a ranking approach to solve this problem. further, @xcite work with data from political debates and model the context of a claim and illustrate the positive impact this has in determining if it has been previously fact-checked. shaar et al. (2021a) publish a dataset and develop a system for detecting all previously fact-checked claims in a lengthy document. text summarization has been used to enable verdict explainability in automatic factchecking @xcite but it hasn't been used for denoising tweets, to the best of our knowledge. @xcite publish the claim-skg knowledge graph, containing 28k factchecked claims and their metadata such as sources, truth value and entities. structured queries can be executed on this knowledge graph, enabling exploration and information discovery. however, it does not provide any mechanism to check if a claim has been previously fact-checked. fact check explorer 9 is a tool developed by google which provides browsing and searching capability for already fact-checked articles which have the claimreview schema 10 embedded. there are performance limitations associated with this tool in the face of long and complex queries. ## dataset the following terms are defined for lucid perusal of this work: a summary of the smc generated using trained models. ## results and discussion this section discusses the results obtained at various stages of our workflow. ## conclusion and future work in this work, a new workflow for detecting previously fact-checked claims is proposed. this workflow uses text summarization as an intermediate step before retrieval module invocation. clean and crisp summaries thus generated are then used for querying a retrieval system. to this end, a first-ofits-kind tweet summarization dataset in the indian context to train such models is curated and released under the cc-by-nc-sa 4.0 license. the performance gained on using popular out-of-the-box and fine-tuned summarization models before querying the fact check explorer is demonstrated, and discussed with qualitative samples. various popular decoding strategies are compared, and the implication of using larger pre-trained models is explored. the aim of this work is to aid in the creation of general-purpose and performant modules which can speed up a fact-checking pipeline by equipping fact-checkers with the tools to fight misinformation at a large scale. in future, we would also like to perform this task in a more general context for news items from various countries, extending our work in a multilingual scenario. also, named entities are crucial in drafting a good query for any retrieval system. generating summaries based on the named entities @xcite found in smcs is a promising avenue to explore. we do not take tweet threads into account as our focus is smcs by users and not replies or comments to those smcs, however, this can be an interesting future direction. other controlled text generation @xcite techniques can also be explored to extract the maximum information from noisy smcs. better pre-training objectives for abstractive summarization on noisy text can lead to efficient outof-the-box models for this task. most indian fact-checking organizations in section 3.1.1 also publish fcas in regional languages such as hindi, tamil and telugu. twitter conversations, spreading misinformation in other pure and transliterated indic languages are voluminous. cross-lingual summarization research @xcite would go a long way in fighting misinformation in a holistic manner.