Are You Serious? Handling Disagreement When Annotating Conspiracy Theory Texts (2024)

Abstract: We often assume that annotation tasks, such as annotating for the presence of conspiracy theories, can be annotated with hard labels, without definitions or guidelines. Our annotation experiments, comparing students and experts, show that there is little agreement on basic annotations even among experts. For this reason, we conclude that we need to accept disagreement as an integral part of such annotations.

## introduction in typical linguistic annotation projects, such as for part of speech tags or for dependency syntax, we assume that there is a single correct analysis, which can be determined reliably by trained annotators (e.g. @xcite . for phenomena such as hate speech or conspiracy theories (cts), we tend to follow the same model without reflecting on the tasks. we assume that we know what hate speech or cts are, asking our annotators to proceed without providing a definition of the phenomenon or guidelines. another way of creating corpora of annotated data consists of using search terms to find examples of the phenomenon, where the search term (or the website from which the example was retrieved) serves as an approximation of the gold standard annotation. for example, a research team from the rand corporation, investigating the automatic detection of "conspiracy theory language" @xcite , used search terms to find examples of ct texts from social media. they created a machine learning classifier for which such texts were to be distinguished from "a baseline sample of 'normal' non-conspiracy talk". the ct texts covered alien visitation, vaccine dangers, the origins of covid-19, while the non-ct texts were sampled from topics such as 'sports', 'movies', 'holidays', etc. similarly, @xcite created a large corpus of ct texts, loco. it provides cts and mainstream texts on the same topics. to gather ct texts, they used a list of ct websites based on scores from mediabiasfactcheck foot_0 . to retrieve mainstream documents, the authors used google to search for the seeds. work by @xcite , who reannotated parts of the loco corpus, shows that the original distinction of conspiracy and mainstream texts is reasonably reliable, but they also found that in some cases, it was difficult to decide whether a ct was perpetuated in a text. note that any bias introduced by annotation or corpus creation will automatically be perpetuated into any machine learning model trained on such a corpus. in the work presented here, we delve deeper into the distinction of ct and mainstream texts. in the first experiment, we asked students to annotate texts to determine how cts were propagated. they were first trained on a set of texts, after having been provided with annotation guidelines. the results showed low inter-annotator agreement (iaa). in a follow-up study, we decided to redo the experiment with ct experts. these results were slightly improved but still did not meet the threshold of acceptable iaa, which raises the question of whether we need better annotation guidelines or whether the ct phenomenon is highly subjective. after a thorough evaluation of the two separate sets of annotations (students and experts), we come to the conclusion that the latter is the case. this suggests that we cannot expect a single gold standard annotation, and consequently cannot use iaa as a measure of the annotation quality. we finally provide a discussion of the consequences of our findings for annotation projects concerned with highly subjective phenomena. our work is closely aligned with other work on annotator disagreement and perspectivist approaches to nlp. thus, our insights are not novel in nlp; we are adding to the discussion by adding conspiracy annotation/detection to the list of subjective tasks, which require perspectivist approaches. ## disagreement in annotation early work on disagreement in linguistic annotations @xcite ) introduced krippendorff's alpha (krippendorff, 2019) as a metric to measure inter-annotator agreement and introduced the notion of explicit and implicit ambiguity in annotations, the latter referring to ambiguity revealed through annotator disagreement. more recent work has started looking into disagreement in annotations beyond measuring it, instead accepting it as a necessary phenomenon in the annotation of subjective tasks for a range of tasks: for example, pos tagging @xcite , textual inference @xcite , sexism @xcite , toxicity @xcite , and hate speech detection @xcite . ## annotating conspiracy theories a ct refers to a story or narrative that claims a small group or 'deep state' has control over the government and is involved in harmful activities aimed at causing widespread harm (e.g. @xcite . this includes doubting scientific evidence of climate change and vaccinations, believing that elections are rigged, and fearing that immigrants, black individuals, or jewish people pose a threat to the rights, freedom, and culture of white people @xcite . empirical evidencebased research from a variety of disciplines seeks to explain why people believe in certain cts (e.g. @xcite @xcite . neville-shepard (2018) and @xcite assert that creators of ct text use language familiar to audiences attracted to conspiracism, and conspiratorial text is grounded in a small amount of 'evidence' that encourages a 'leap of faith' by the audience to reach conclusions. this suggests that conspiratorial discourse caters to people in the know and, therefore, does not explicitly convey a premise, rather, the audience completes the argument based on prior knowledge. @xcite further assert that creators of ct text depend on audiences already familiar with similar ideas. this means that these theories tap into a broader culture of belief in conspiracies and act as a way for people to find others who share their beliefs and reinforce their convictions. however, this makes it difficult to classify conspiratorial text because, without an explicit claim, it is up to the audience to 'leap' to conclusions based on the familiar tropes presented in the text. @xcite have reannotated parts of the loco corpus @xcite to determine how reliable the automatic corpus collection was. they started with a simple definition of what they considered a ct text, namely a text that propagated a conspiracy belief, defined as: "a conspiracy belief is the belief that an organization made up of individuals or groups was or is acting covertly to achieve some malevolent end. it depicts causal narratives of an event as a covert plan orchestrated by a secret cabal of people or organizations instead of a random or natural happening" @xcite . their first round of annotations showed high iaa for mainstream texts, but the iaa for ct was 0.47. as a consequence, they adjusted the guidelines and added that in order for a text to be considered a ct text, the following had to hold: "a document is considered ct if and only if such a belief is manifested in the text via specific expressions" @xcite . they also identified a set of textual and verbal cues that triggered a reading of conspiracy, e.g., all caps texts, paraphrases, questions. the revised guidelines resulted in a higher iaa for conspiracy texts of 0.70. however, when they used the same annotation scheme for a different conspiracy theory, the results for ct texts was considerably lower (0.58), thus showing that robust annotations are difficult, even with trained annotators. ## the annotation study we conducted an annotation study to determine which circumstances (in terms of training) we need to reliably annotate a range of phenomena. more specifically, the annotations covered identifying similarities in main themes, structures, rhetorical forms, and tropes. for the annotation samples, we revisited the loco corpus to draw a sample of cts. we only extracted documents identified as a conspiracy, representing a broad range of topics, using the seeds covid-19, pizzagate, climate change, jfk assassination, 9/11, illuminati, and flat earth. our non-experts are graduate students without prior knowledge of the litera-ture on cts. the experts are researchers who have worked on cts for at least 2 years. we report both fleiss' kappa @xcite ) and krippendorff's alpha @xcite . the annotation scheme is based on past research on hate speech, ct text, and populist rhetoric @xcite @xcite . we used the following questions: we first trained the non-expert annotators on a sample of 11 conspiracy documents selected from the seven cts (covid-19, pizzagate, climate change, jfk assassination, 9/11, illuminati, and flat earth). after training, they independently annotated a random sample of 472 ct texts representing the same conspiracy topics. the annotations were conducted by two ma students and three phd students without prior knowledge of research on cts. the results are shown in as part of this second round of annotations, we added questions to capture experts' certainty of their annotation for items flagged controversial (e.g., very certain, pretty sure, not sure, i have no idea/guessed). the answers show that while the experts never guessed, the majority were only pretty sure (61-84%) or not sure (3-18%). due to the lack of agreement for non-experts and experts, we decided to simplify the coding scheme to the first three questions, but to clarify and extend the guidelines, to see if more explicit instructions would increase iaa. the modifications were based on a discussion of the experts of which uncertainties they faced during the annotation process. we used another sub-sample of 10 cts from the same sample the students annotated. the results are based on 6 expert annotators and are shown in during the last experiment, we also asked the larger group of experts to describe any difficulty they had determining the answers. we show sample responses in ## consequences for annotation projects our results above show clearly that it is extremely difficult to reach high iaa on even basic questions such as whether a text contains a ct, even when experts are used as annotators. it is pos-q1 text read as if an excerpt from a news story [text 1] q1 the overall passage read as if not true, but hard to discern a specific ct q1 unsure if this is a ct or simple a dispute [text 2] q2 sounds like a movie plot [text 2] q2 this one was difficult because it's describing a real thing that happened in language that's a bit bombastic, and also acknowledges an offshoot that may or may not actually exist. [text 3] q2 illuminati is mentioned, but the main text assumes the reader knows the illuminati are perpetrating mind control and other atrocities. q3 needs fact checking q3 narrative was about combating the spread of covid-19 [text 1] q3 unsure about this one -it mentions misinformation but is it a ct? q3 it engages with cts and seems to endorse them, but it is more about getting you to pay attention and stay. a closer look at the texts and the annotations allows us to conclude that the annotations are and need to be subjective. we cannot have a single gold standard annotation; rather, we must be prepared to accept a range of answers. this conclu-sion leads to several consequences, partly for the definition of the problem, and partly for modeling the problem computationally. 1. disagreement between annotators is not a sign of lack of clarity in the annotation scheme but a direct consequence of the phenomenon to be annotated. one reason is that belief in cts is not monolithic, but rather faceted, where individuals belief in subsets of factoid of a range of cts. another reason is that we need to model the preception of standard readers, and their interpretation will depend on prior knowledge as well as prior bias. ## limitations our comparative annotation study is based on a small number of annotators since it was supposed to serve as a pilot stud for a larger annotation project. however, the students went through a thorough training session, and the number of experts is naturally limited by availability. all expert annotators are also co-authors on this report, our expertise ranges over a wide array of fields, which ensures a wide disciplinary stance. ## ethical considerations working with cts tends to be difficult for the annotators. for this reason, we concentrated on a set of different cts that are less prone to explicit hatred that is often present in cts targeting specific minority groups (e.g., white genocide). however, despite this careful selection, the chosen texts can contain content that may upset annotators. a appendix text 1: mainland china reported 30 new coronavirus cases on saturday, up from 19 a day earlier as the number of cases involving travellers from abroad as well as local transmissions increased, highlighting the difficulty in stamping out the outbreak. the national health commission said in a statement on sunday that 25 of the latest cases involved people who had entered from abroad, compared with 18 such cases a day earlier. five new locally transmitted infections were also reported on saturday, all in the southern coastal province of guangdong, up from a day earlier. the mainland has now reported a total of 81,669 cases, while the death toll has risen by three to 3,329. though daily infections have fallen dramatically from the height of the epidemic in february, when hundreds of new cases were reported daily, beijing remains unable to completely halt new infections despite imposing some of the most drastic measures to curb the virus spread. the so-called imported cases and asymptomatic patients, who have the virus and can give it to others but show no symptoms, have become among chinas chief concerns in recent weeks. the country has closed off its borders to almost all foreigners as the virus spread globally, though most of the imported cases involve chinese nationals returning from overseas. the platinum standard of advanced multivitamin formulations is back in stock! order vitamin mineral fusion at 50% off with double patriot points and free shipping today! text 2: between 542 and 66 million years ago -long before the "supervolcano" became part of yellowstone's geologic story -the area was covered by inland seas. nps/jim peaco most of earth's history (from the formation of the earth 4.6 billion years ago to approximately 541 million years ago) is known as the precambrian time. rocks of this age are found in northern yellowstone and in the hearts of the teton, beartooth, wind river, and gros ventre ranges. during the precambrian and the subsequent paleozoic and mesozoic eras (541 to 66 million years ago), the western united states was covered at times by oceans, sand dunes, tidal flats, and vast plains. from the end of the mesozoic through the early cenozoic, mountain-building processes formed the rocky mountains. during the cenozoic era (approximately the last 66 million years of earths history), widespread mountain-building, volcanism, faulting, and glaciation sculpted the yellowstone area. magma (molten rock from earth's mantle) has been close to the surface in yellowstone for more than 2 million years. its heat melted rocks in the crust, creating a magma chamber of partially molten, partially solid rock. heat from this shallow magma caused an area of the upper crust to expand and rise. the yellowstone plateau became a geomorphic landform shaped by episodes of volcanic activity. stress also caused rocks overlying the magma to break, forming faults and causing earthquakes. eventually, these faults reached the deep magma chamber. magma oozed through these cracks, releasing pressure within the chamber and allowing trapped gases to expand rapidly. a massive volcanic eruption then occurred along vents, spewing volcanic ash and gas into the atmosphere and causing fast super-hot debris (pyroclastic) flows on the ground. as the underground magma chamber emptied, the ground above it sunk, creating the first of yellowstones three calderas. this diagram shows the general ideas behind two theories of how magma rises to the surface. adapted with permission from windows into the earth by robert smith and lee j. siegel, 2000. researchers found that the changes leading up to an eruption may happen in a matter of decades rather than thousands of years in advance as previously thought. based on minerals from the last major eruption, the supervolcanoes are characterized as volcanic centers that have had eruptions that covered more than 240 cubic miles. the us has two: one in yellowstone and another in californias long valley. an eruption could emit ash that would expand over 500 miles. the eruption would likely cover the ground with as much as 4 inches of gray ash, which could be detrimental to crops growing in the midwest. another less worrisome concern is the 1,000 degree f molten lava that could ooze out. gases, including sulfur dioxide, which contributes to acid rain would be spewed from the supervolcano and the global cooling issues associated with reflecting sunlight away from the earth are also concerns. but there are other supervolcanos in the world with sooner predictions than yellowstones. campi flegri, a name that aptly translates as burning fields, is in a critical state, according to researchers in italy. it consists of a vast and complex network of underground chambers that formed hundreds of thousands of years ago, stretching from the outskirts of naples to underneath the mediterranean sea. though its last eruption was in 1538, its due for an eruption soon. it would be a minor event compared to the 72 cubic miles of molten rock it spewed in its most notorious eruption 39,000 years ago, called campanian ignimbrite, that likely contributed to the extinction of the neanderthals. fortune website article reported that if the yellowstone supervolcano erupts, it could shoot out more than 1,000 cubic kilometers of rock and ash into the air. thats 250 cubic miles. thats more than three times as large as the campanian ignimbrite eruption in italy, which created a sulfurous cloud that floated more than 1,200 miles away to hang over russia. thats 2,500 times more material than mount st. helens expelled in 1980, killing 57 people. an eruption at yellowstone would result in a cloud of ash more than 500 miles wide, stretching across nearly the entire western united states. nasa has a plan to neutralize supervolcano threats however. they believe the most viable solution could be to drill up to 6 miles down into the supervolcano, and pump down water at high pressure. the circulating water would return at a temperature of around 662f, thus slowly day by day extracting heat from the volcano. and while such a project would come at an estimated cost of around $3.46 billion, it comes with an enticing catch which could convince politicians to make the investment. it would become a source of geothermal energy. but there are considerable risks, too. it could trigger the eruption its meant to save us from. historically, four types of volcanic events have taken place in yellowstone (you may click on each one to learn more): the likelihood of an eruption in the near future is still low. however those who instigate such a project will never see it to completion, or even have an idea whether it might be successful within their lifetime. cooling yellowstone in this manner would happen at a rate of 3.2808399 feet a year, taking of the order of tens of thousands of years until just cold rock was left. featured image: yellowstone harbours a giant magma chamber that will blow one day if we dont act (credit: istock) the victim was kept in a chemically induced sleep for weeks and subjected to rounds of electroshocks, experimental drugs and tape-recorded messages played non-stop.