Structurizing Misinformation Stories via Rationalizing Fact-Checks (2021)

Abstract: Misinformation has recently become a well-documented matter of public concern. Existing studies on this topic have hitherto adopted a coarse concept of misinformation, which incorporates a broad spectrum of story types ranging from political conspiracies to misinterpreted pranks. This paper aims to structurize these misinformation stories by leveraging fact-check articles. Our intuition is that key phrases in a fact-check article that identify the misinformation type(s) (e.g., doctored images, urban legends) also act as rationales that determine the verdict of the fact-check (e.g., false). We experiment on rationalized models with domain knowledge as weak supervision to extract these phrases as rationales, and then cluster semantically similar rationales to summarize prevalent misinformation types. Using archived fact-checks from Snopes.com, we identify ten types of misinformation stories. We discuss how these types have evolved over the last ten years and compare their prevalence between the 2016/2020 US presidential elections and the H1N1/COVID-19 pandemics.

## introduction misinformation has raised increasing public concerns globally, well-documented in africa @xcite , asia @xcite , and europe @xcite . in the us, "fake news" accounted for 6% of all news consumption during the 2016 us presidential election @xcite . years later, 29% of us adults in a survey believed that the "exaggerated threat" of the covid-19 pandemic purposefully damaged former us president donald trump @xcite , and 77% of trump's supporters believed "voter fraud" manipulated the 2020 us presidential election in spite of a complete lack of evidence @xcite . each line is a snippet from a fact-check. key phrases identifying the misinformation types are highlighted. phrases with similar semantics are clustered in colored boxes. this structure is a sample of our final results. as such misinformation continues to threaten society, researchers have started investigating this multifaceted problem, from understanding the socio-psychological foundations of susceptibility @xcite and measuring public responses @xcite , to designing detection algorithms @xcite and auditing countermeasures for online platforms @xcite . these studies mostly adopted the term "misinformation" as a coarse concept for any false or inaccurate information, which incorporates a broad spectrum of misinformation stories, e.g., political conspiracies to misinterpreted pranks. although misinformation types have been theorized and categorized by practitioners @xcite , there is, to our knowledge, no empirical research that has systematically measured these prevalent types of misinformation stories. this paper aims to unpack the coarse concept of misinformation and structurize it to fine-grained story types (as illustrated in figure 1 ). we conduct this query through an empirical lens and ask the question: what are the prevalent types of misinformation stories in the us over the last ten years? the answer to our question is buried in archived fact-checks, which are specialized news articles that verify factual information and debunk false claims by presenting contradictory evidence @xcite . as a critical component of their semi-structured journalistic style, fact-checks often embed the (mis)information type(s) within their steps of reasoning. for example, consider the following snippet from a snopes.com fact-check with a verdict of false @xcite : "...for instance, some started sharing a doctored photograph of thunberg with alt-right boogeyman george soros (the original photograph featured former vice president al gore)..." the key phrase doctored photograph in the snippet identifies the misinformation type of the factchecked story. additional example phrases are highlighted in figure 1 . with a large corpus of fact-checks, these phrases would accumulate and reveal prevalent types of misinformation stories. extracting these phrases is a computational task. our intuition is that such phrases in a fact-check also act as rationales that determine the verdict of the fact-check. in the previous example, the verdict is false in part because the story contains a doctored photograph. therefore, a neural model that predicts the verdict of a fact-check would also use the misinformation types as rationales. to realize this intuition, we experiment on existing rationalized neural models to extract these phrases @xcite , and, to target specific kinds of rationales, we additionally propose to include domain knowledge as weak supervision in the rationalizing process. using public datasets as validation @xcite , we evaluate the performance variation of different rationalized models, and show that including domain knowledge consistently improves the quality of extracted rationales. after selecting the most appropriate method, we conduct an empirical investigation of prevalent misinformation types. using archived fact-checks from snopes.com, spanning from its founding in 1994 to 2021, we extract rationales by applying the selected model with theorized misinformation types for weak supervision @xcite , and then cluster rationales based on their semantic similarity to summarize prevalent misinformation types. we identify ten types of misinformation stories, a preview of which are shown in figure 1 . using our derived lexicon of these clustered misinformation stories, we then explore the evolution of misinformation types over the last ten years. our key findings include: increased prevalence of conspiracy theories, fabricated content, and digital manipulation; and decreased prevalence of legends and tales, pranks and jokes, mistakes and errors, etc. we also conducted two case studies on notable events that involve grave misinformation. from the case study of us presidential elections, we observe that the most prevalent misinformation type for both the 2016 and 2020 elections is fabricated content, while the 2016 election has more hoaxes and satires. from the case study of pandemics, our results show that the h1n1 pandemic in 2009 has more legends and tales, while the covid-19 pandemic attracts more conspiracy theories. the code and data used in the paper are available at: https://factcheck.shanjiang.me . ## related work there is a rich literature that has studied the online misinformation ecosystem from multiple perspectives @xcite . within the computational linguistics community, from an audiences' perspective, @xcite found that social media users expressed different linguistic signals when responding to false claims, and the authors later used these signals to model and measure (dis)beliefs in (mis)information @xcite . from a platforms' perspective, researchers have assisted platforms in designing novel misinformation detection methods @xcite @xcite , as well as audited existing misinformation intervention practices @xcite @xcite . in this work, we study another key player in the misinformation ecosystem, storytellers, and investigate the prevalent types of misinformation told to date. from the storytellers' perspective, @xcite theorized several potential misinformation types (e.g., satire or parody, misleading content, and false connection), yet no empirical evidence has been connected to this typology. additionally, researchers have investigated specific types of mis-information as case studies, e.g., state-sponsored disinformation @xcite , fauxtography @xcite , and conspiracy theories @xcite . in this paper, we aim to structurize these misinformation stories to theorized or novel types. ## rationalized neural models realizing our intuition (as described in § 1) requires neural models to (at least shallowly) reason about predictions. in this section, we introduce existing rationalized neural models and propose to include domain knowledge as weak supervision in the rationalizing process. we then experiment with public datasets and lexicons for evaluation. ## rationalizing fact-checks after determining that soft rationalization is the most appropriate method, we apply it to extract rationales from fact-checks. in this section, we introduce the dataset we collected from snopes.com and conduct experiment with fact-checks to structurize misinformation stories. ## evolution of misinformation in this section, we leverage the clusters of misinformation types identified by our method as a lexicon and apply it back to the our original fact-check dataset. specifically, we analyze the evolution of misinformation types over the last ten years and compare misinformation trends around major realworld events. evolution over the last ten years. we first explore the evolution of misinformation over time. we map each fact-check article with one or more corresponding misinformation types identified by our method, and then aggregate fact-checks by year from before 2010 foot_10 to the end of 2020 to estimate the relative ratio of each misinformation type. as shown in figure 4 , 12 the prevalence of certain misinformation types on snopes.com has drastically changed over the last ten years. heavily politicized misinformation types, such as digitally altered or doctored images or photographs , fabricated and misleading content , and conspiracy theories have nearly doubled in relative ratios over the last ten years. in contrast, the prevalence of (arguably) less politicized stories, such as legends and tales , hoaxes and pranks , and mistakes and errors have decreased. these trends may be a proxy for the underlying prevalence of different misinformation types within the us. studies that measure political ideologies ## vs. 2020 us presidential election. we now compare misinformation types between the 2016 and 2020 elections. to filter for relevance, we constrain our analysis to fact-checks that (1) were published in the election years and (2) included the names of the presidential candidates and/or their running mates (e.g., "joe biden" and "kamala harris"). this results in 2,586 fact-checks for the 2016 election and 2,436 fact-checks for 2020. the prevalence of each misinformation type is shown in figure 5 . we observe that the relative ratios of many misinformation types are similar between the two elections, e.g., legends and tales and bogus scams , while the 2016 election has more hoaxes , satires , etc. the most prevalent type during both elections is fabricated and misleading content , next to conspiracy theories . h1n1 vs. covid-19. finally, we compare misinformation types between the h1n1 pandemic in 2009 and the covid-19 pandemic. for h1n1 related fact-checks, we search for keywords "flu", "influenza", and "h1n1" in fact-checks and constrain the publication date until the end of 2012. 14 for covid-19 related fact-checks, we search for keywords "covid-19" and "coronavirus", and only consider fact-checks published in 2019 or later, which results in 833 fact-checks for the h1n1 pandemic and 656 fact-checks for covid-19. the relative ratio of each misinformation type is also shown in figure 5 . we observe that the prevalence of some misinformation types are sig- ## discussion in this section, we discuss limitations of our work and future directions, and finally conclude. limitations and future directions. we adopted a computational approach to investigate our research question, and this method inherently shares common limitations with observational studies, e.g., prone to bias and confounding @xcite . specifically, our corpus contains fact-checks from snopes.com, one of the most comprehensive fact-checking agencies in the us. snopes.com covers a broader spectrum of topics than politics-focused fact-checkers (e.g., politi-fact.com, factcheck.org), 15 and thus we argue that it covers a representative sample of misinformation within the us. however, snopes.com may not be representative of the international misinformation ecosystem @xcite @xcite . in the future, we hope that our method can help characterize misinformation comparatively on a global scale when more structured fact-checks become available. 16 additionally, fact-checkers are time constrained, as thus the misinformation stories they cover tend to be high-profile. therefore low-prevalence, long-tail misinformation stories may not be observed in our study. understanding low-volume misinformation types may require a different collection of corpora other than fact-checks, e.g., a cross-platform investigation on social media conversations @xcite . lastly, the misinformation types we extract from our weakly supervised approach are not validated with ground-truth labels. this is largely due to the lack of empirical knowledge on misinformation types, and therefore we are unable to provide specific guidance to annotators. although the clusters in figure 3 provide straightforward structure of misinformation stories, in future work, we plan to leverage these results to construct annotation guidelines and obtain human-identified misinformation types for further analysis.