From Chaos to Clarity: Claim Normalization to Empower Fact-Checking (2023)

Abstract: With the proliferation of social media platforms, users are exposed to vast information, including posts containing misleading claims. However, the pervasive noise inherent in these posts presents a challenge in identifying precise and prominent claims that require verification. Extracting the core assertions from such posts is arduous and time-consuming. We introduce a novel task, called Claim Normalization ( \textit{aka ClaimNorm} ) that aims to decompose complex and noisy social media posts into more straightforward and understandable forms, termed \textit{normalized claims} . We propose \texttt{CACN} , a pioneering approach that leverages chain-of-thought and claim check-worthiness estimation, mimicking human reasoning processes, to comprehend intricate claims. Moreover, we capitalize on large language models’ powerful in-context learning abilities to provide guidance and improve the claim normalization process. To evaluate the effectiveness of our proposed model, we meticulously compile a comprehensive real-world dataset, \texttt{CLAN} , comprising more than 6k instances of social media posts alongside their respective normalized claims. Experimentation demonstrates that \texttt{CACN} outperforms several baselines across various evaluation measures. A rigorous error analysis validates \texttt{CACN} ‘s capabilities and pitfalls. We release our dataset and code at https://github.com/LCS2-IIITD/CACN-EMNLP-2023.

## introduction social media have enabled a new way of communication, breaking down geographical barriers and bringing unprecedented opportunities for knowledge exchange. however, this has also presented a growing threat to society, e.g., during the 2016 us presidential election @xcite , the covid-19 pandemic @xcite @xcite , the ukraine-russia conflict @xcite , etc. false claims are an intrinsic aspect of fabricated news, rumors, propaganda, and misinformation. journalists and fact-checkers work tirelessly to assess the factuality of such claims in spoken and/or written form, sifting through an avalanche of claims and pieces of evidence to determine the truth. to further address this pressing issue, several independent fact-checking organizations have emerged in recent years, such as snopes, 2 fullfact, 3 and poli-tifact, 4 which play a crucial role in verifying the accuracy of online content. however, the rate at which online information is being disseminated far outpaces the capacity of fact-checkers, making it difficult to verify every single claim. this, in turn, leaves numerous unverified claims circulating online, potentially reaching millions before they can be verified. while the complete automation of the factchecking pipeline may pose hazards to accountability and reliability, several recent studies have targeted identifying downstream tasks suitable for automation, such as detecting claims @xcite @xcite , evaluating their worthiness for factchecking @xcite @xcite , making sure they were not fact-checked before @xcite @xcite , and validating them by retrieving relevant shreds of evidence @xcite @xcite . see also a recent survey on automated fact-checking for assisting human fact-checkers @xcite . in light of the growing challenges faced by factcheckers in verifying the factuality of social media claims, we propose the novel task of claim normalization. this task aims to extract and to simplify the central assertion made in a long, noisy social media post. this can improve the efficacy and curtail the workload of fact-checkers while maintaining high precision and conscientiousness. we provide a more detailed explanation of why the claim normalization task is essential and illustrate its significance in appendix a.1. in our problem formulation, given an input social media claim, the system needs to simplify it in a concise form that contains the post's central assertion that fact-checkers can easily verify. to better understand our motivation, we illustrate the task in figure 1 . the first social media post reads, 'cyanocobalamin is a synthetic form of vitamin b12...if you're on b12 supplements, throw them away.' this post contains some extraneous information that has no relevance for fact-checkers. as a result, they distil the information and summarize it as, 'cyanocobalamin, the most common form of vitamin b12, is toxic.' fact-checkers tasked with verifying the accuracy of such noisy posts need to read through them and condense their content to obtain a concise claim that can be easily fact-checked. unfortunately, this process can be exceedingly timeconsuming. by automating the claim normalization process, fact-checkers can work more efficiently. another aspect is that fact-checkers often choose what to fact-check based on the virality of a claim, for which they need to be able to recognize when the same claim appears in a slightly different form, and claim normalization is essential for this. our contributions are as follows: ## related work claim analysis. previous work has focused on distinct aspects of claims, including claim detection @xcite @xcite , claim check-worthiness estimation @xcite @xcite @xcite @xcite , claim span identification @xcite , etc. by curating the aawd corpus, @xcite pioneered the efforts in claim detection, the foremost step in the fact-checking tasks. following this, linguistically motivated features, including sentiment, syntax, context-free grammar, and parse trees, were frequently used @xcite @xcite . recently, large language models (llms) have also been used for claim detection @xcite @xcite . most previous work on claim detection and extraction primarily concentrated on adapting to text that comes from similar distributions or topics. moreover, it often relied on well-structured formal writing. in contrast, our objective is to develop a system that specifically addresses the challenges posed by posts in social media and aims to extract the central claim in a more simplified manner, which goes beyond extracting a text subspan in a social media post and aims at abstractive claim extraction that mimics what professional fact-checkers do. to the best of our knowledge, we are the first to address the task of claim extraction in this very practical formulation. text summarization. the task of claim normalization is closely related to the task of text summarization. in the latter, given a lengthy document, the goal is to summarize it into a much shorter summary. previous work on text summarization has explored various approaches, including large pretrained seq2seq models to generate high-quality summaries @xcite @xcite . one issue has been the faithfulness of the summary with respect to the source. to address this, @xcite introduced factcc, a weakly-supervised bert-based entailment model, which augments the dataset with artificially introduced faithfulness errors. similarly, @xcite trained a model for detecting factual inconsistencies in data from controllable text generation that perturbs human-annotated summaries, introducing varying types of factual inconsistencies. @xcite proposed a questionanswering framework that compares answers from the summary to those from the original text. all these approaches primarily focused on general-purpose summarization and did not provide means for models to generate summaries primarily focusing on specific needs. to address this limitation, controlled summarization was introduced @xcite . one aspect of controlled summarization is length control, in which users can set their preferred summary length @xcite . recent research has discovered that, despite their fluency and coherence, state-of-the-art abstractive summarization systems produce summaries with contradictory information. while text summarization systems can assist in condensing social media posts into shorter summaries, their primary goal is not to ensure verifiability. it aims to capture the key points of the text rather than emphasizing the specific claims within the text that need to be fact-checked. our task of claim normalization, on the other hand, works at an entirely different level. it needs a thorough understanding of the claims made in the social media post and strives to ensure that the normalized claims are not only consistent with the original post, but are also self-contained and verifiable. despite the progress in text summarization, the task of claim normalization remains underexplored. in this work, we aim to tackle this challenging problem by developing a robust approach specifically tailored to the unique aspects of this task. ## dataset existing text summarization datasets have not specifically addressed the need for claim-oriented summaries. to address this gap, we propose a novel dataset clan (claim normalization), consisting of fact-checked social media posts paired with concise claim-oriented summaries (known as normalized claims), created by fact-checkers as part of the verification process. as a result, our dataset is not subjected to external annotation, thus averting potential biases and ensuring its high quality. ## proposed approach in this section, we explain our proposed approach, check-worthiness aware claim normalization (cacn), which aims to integrate task-specific information with large language models (llms). we focus our experiments on gpt-3 (text-davinci-003) @xcite . our approach amalgamates two key ideas: (i) chain-of-thought prompting and (ii) reverse check-worthiness. ## experimental setup baseline models. for comparison, we use several state-of-the-art generative systems and categorize them into two groups: (i) pre-trained large language models (plms): t5 (raffel et al., 2020), bart @xcite , flan-t5 (chung et al., 2022), and pegasus @xcite . for t5, bart, and flan-t5, we use base and large model sizes. for pegasus, we use the reddit model. (ii) in-context learning model: gpt-3 (text-davinci-003) @xcite . ## experiments and evaluation our experiments reveal that our cacn outperforms all baselines across most evaluation measures. we further examine all systems aiming to answer the research questions listed below. do meticulously crafted prompts enhance the performance of generative models? the findings exhibit a significant performance improvement when using prompt-tuning, specifically with incontext examples. is training models on a specific task less effective than in-context learning with a few examples? we observe substantial disparities in the performance of models trained on task-specific data compared to using in-context learning with a limited number of examples, as shown in we can see that the models exposed to in-context examples showcase superior performance, highlighting their efficacy in capturing task-specific patterns. while the trained models exhibit excellence in lexical metrics, their performance in semantic metrics is noticeably lower. notably, bart large , trained on our dataset, outperforms other trained models by sizable margins. these results strongly underline that, within the realm of llms, incorporating prompt-tuning with incontext learning holds more promise, leading to enhanced generalization capabilities. do models demonstrate inherent proficiency in generating normalized claims with minimal or no prior training? we examine the potential benefits of zero-shot and few-shot learning to investigate the inherent proficiency in generating normalized claims. the zero-shot and the few-shot results are shown in ## qualitative analysis error analysis. to comprehend the performance of cacn, we strive to qualitatively analyze the errors committed by our model in this section. naturally, the predictions in the fine-grained analysis are much more intricate than in the coarsegrained quantitative setup. during our manual qualitative analysis, we unveiled several interesting patterns and errors in the generated responses. for example, although bart large generated responses with a high bertscore in example 1, we noticed that the factual alignment is incorrect, making this model untrustworthy for downstream tasks such as claim check-worthiness and claim verification. in contrast, our proposed model produced a response that is both correct and precise. the response generated by direct is also accurate, but it is excessively long, which contradicts the objective of the normalized claims being concise and straightforward. this problem is also evident in example 3, where direct produces a factually correct claim but is overly long. in example 2, we observe that the bart large model demonstrates the lowest number of hallucinations and adheres closely to the input social media post. in contrast, our model's bertscore performed the worst for this example. however, upon closer inspection, we noticed that the normalized claim that our model generated was indeed correct and most relevant for fact-checking. these findings highlight the complexity and the trade-offs involved in generating normalized claims. while certain models may excel in certain cases, there is often a compromise in other aspects, such as factual accuracy and conciseness. ## conclusion and future work we introduced the novel task of claim normalization, which holds substantial value on multiple fronts. for human fact-checkers, claim normalization is a useful tool that can assist them in effectively removing superfluous texts from subsequent processing. this also benefits downstream tasks such as identifying previously fact-checked claims, estimating claim check-worthiness, etc. we further compiled a dataset of social media posts comprising over 6k posts and their normalized claims. we further benchmarked this dataset with a novel approach, cacn, and showed its superior performance compared to different state-of-the-art generative models across multiple assessment measures. we also documented our data collection process, providing valuable insights for future research in this domain. in future work, we plan to extend the dataset, including with new languages. we also plan to use more powerful llms.