Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID -19 Treatments (2023)

Abstract: We present a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. Our approach extracts check-worthy claims, which are aggregated and ranked for review. Stance classifiers are then used to identify tweets supporting novel misinformation claims, which are further reviewed to determine whether they violate relevant policies. To demonstrate the feasibility of our approach, we develop a baseline system based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. We make our data and detailed annotation guidelines available to support the evaluation of human-in-the-loop systems that identify novel misinformation directly from raw user-generated content.

## introduction as many people now get information from social networking websites such as facebook and twitter, misinformation has become a serious societal problem. to address this, social media companies have spent billions on content moderation. 2 prior work on developing natural language processing systems to combat misinformation has mainly focused on various sub-tasks @xcite , including claim detection @xcite , evidence retrieval @xcite @xcite , fact verification @xcite @xcite , stance classification @xcite @xcite , and fallacy recognition @xcite . researchers have also attempted to perform early detection of novel misinformation claims @xcite , as it is crucial for supporting early interventions such as pre-bunking @xcite . however, evaluations are often set up automatically using datasets that were retrospectively constructed based on a predefined set of debunked claims. recent work by @xcite presented convincing evidence that existing nlp factchecking pipelines are unsuitable for detecting novel real-world misinformation. they show these systems rely on leaked counter-evidence from news sources that have already fact-checked the claim. in general, it is unrealistic to assume this type of evidence will be available for new claims that have not yet been widely spread. in this paper, we address this challenge by presenting a more realistic human-in-the-loop detection and evaluation framework that can measure a system's capabilities for detecting novel checkworthy claims in the wild (see figure 1 ). we focus on discovering new, domain-specific claims from raw tweets which are then verified by humans, rather than relying on a pre-defined list of claims that have already been fact-checked for evaluation. more importantly, we consider not only the accuracy but also the volume, relevance, and timeliness of misinformation claims automatically identified by a system, given a collection of raw tweets. we argue this approach provides more realistic experimental conditions because (1) it does not rely on leaked counter-evidence from claims that have already been fact-checked, (2) human expertise is vital in verifying the truthfulness of claims @xcite and (3) it is more effective for humans to check aggregated claims within a specific domain (e.g., claims about the efficacy of covid-19 treatments), before proceeding to individual social media messages to determine if they violate specific misinformation policies. we validate our methodology for end-to-end misinformation detection in the domain of trending claims @xcite raw tweets treatments. covid-19 treatments make an ideal testbed for human-in-the-loop misinformation extraction because twitter has provided clearly defined policies in this area, which we use as guidelines in a realistic human evaluation of a system's output. 3 we evaluate our baseline system with our four defined metrics and find that 18% of the top-50 trending claims were actually misleading (relevance), 50% of new misleading claims (unapproved covid-19 treatments) are detected before they are debunked by journalists in a news article (timeliness), 65% of tweets flagged constitute policy-violations (accuracy), and an average of 124 policy violations can be confirmed by a human-annotator per hour (volume) when using our system. our work fills an important gap in the literature, by showing that it is possible to construct a realistic end-to-end evaluation that supports the early detection of novel rumors directly from raw data. instead of classifying individual tweets as rumorous or not, we extract phrase-level claims that can be aggregated and ranked across a large amount of data and thus can be reviewed more time-efficiently by fact-checkers for human evaluation and for realworld applications. tweets that are automatically classified as supporting misinformation claims can then be reviewed to determine whether they violate relevant policies. ## related work there is a large body of misinformation-related research. due to space limitations, we only highlight the most relevant work. see also the excellent surveys by @xcite and @xcite . ## human-in-the-loop evaluation framework one of the most important functions of a misinformation detection system is to identify new misinformation claims in the wild, and in a timely manner. we thus design our evaluation framework to measure not only the accuracy but also the volume, relevance, and timeliness of misinformation claims identified by a system, given a large collection of raw tweets (not collected based on already debunked claims). see figure 1 for an overview of our framework. ## a case study: covid-19 treatment misinformation to illustrate the usage of our human-in-the-loop evaluation framework outlined in §3, we present a case study for covid-19 misinformation. specifically, we target twitter's covid-19 policy on unapproved treatments, which states that: "false or misleading information suggesting that unapproved treatments can be curative of covid-19" are grounds for labeling tweets with corrective information @xcite . ## conclusion in this work, we present a novel end-to-end humanin-the-loop evaluation framework for the early identification of novel misinformation on social media from raw tweets. unlike previous evaluation frameworks, our methodology captures the interplay between the system and human content moderators while also providing realistic metrics for early misinformation detection. we validate our misinformation detection framework for claims in the domain of covid-19 treatments. by aggregating and ranking structured representations of claims, and relying on human fact-checkers to review trending claims, our system is able to detect 50% of misleading claims earlier than the news.