Which side are you on? Insider-Outsider classification in conspiracy-theoretic social media (2022)

Abstract: Social media is a breeding ground for threat narratives and related conspiracy theories. In these, an outside group threatens the integrity of an inside group, leading to the emergence of sharply defined group identities: Insiders – agents with whom the authors identify and Outsiders – agents who threaten the insiders. Inferring the members of these groups constitutes a challenging new NLP task: (i) Information is distributed over many poorly-constructed posts; (ii) Threats and threat agents are highly contextual, with the same post potentially having multiple agents assigned to membership in either group; (iii) An agent’s identity is often implicit and transitive; and (iv) Phrases used to imply Outsider status often do not follow common negative sentiment patterns. To address these challenges, we define a novel Insider-Outsider classification task. Because we are not aware of any appropriate existing datasets or attendant models, we introduce a labeled dataset (CT5K) and design a model (NP2IO) to address this task. NP2IO leverages pretrained language modeling to classify Insiders and Outsiders. NP2IO is shown to be robust, generalizing to noun phrases not seen during training, and exceeding the performance of non-trivial baseline models by 20%.

## background and motivation narrative models -often succinctly represented as a network of characters, their roles, their interactions (syuzhet) and associated time-sequencing information (fabula) -have been a subject of considerable interest in computational linguistics and narrative theory. stories rest on the generative backbone of narrative frameworks @xcite . while the details might vary from one story to another, this variation can be compressed into a limited set of domain-dependent narrative roles and functions @xcite . social narratives that both directly and indirectly contribute to the construction of individual and group identities are an emergent phenomenon resulting from distributed social discourse. currently, this phenomenon is most readily apparent on social media platforms, with their large piazzas and niche enclaves. here, multiple threat-centric narratives emerge and, often, over time are linked together into complex conspiracy theories @xcite . conspiracy theories, and their constituent threat narratives (legend, rumor, personal experience narrative) share a signature semantic structure: an implicitly accepted insider group; a diverse group of threatening outsiders; specific threats from the outsider directed at the insiders; details of how and why outsiders are threatening; and a set of strategies proposed for the insiders to counter these threats @xcite . indeed, the insider/outsider groups are fundamental in most studies of belief narrative, and have been exhaustively studied in social theory and more specifically, in the context of conspiracy theories @xcite . on social media, these narratives are negotiated one post at a time, expressing only short pieces of the "immanent narrative whole" @xcite . this gives rise to a new type of computational linguistic problem: given a large enough corpus of social media text data, can one automatically distill semantically-labeled narratives (potentially several overlapping ones) that underlie the fragmentary conversational threads? recent work @xcite @xcite has shown considerable promise that such scalable automated algorithms can be designed. an automated pipeline of interlocking machine learning modules decomposes the posts into actors, actants and their inter-actant relationships to create narrative networks via aggregation. these network representations are interpretable on inspection, allowing for the easy identification of the various signature semantic structures: insiders, colored spans are used to highlight noun phrases that are inferred (red for outsiders; blue for insiders). pos tags are shown along with the noun phrases to illustrate an example of syntactic and semantic hints used by np2io to generate the inferred labels. note that, based solely on context, the same agents ("tech", "vaccines", "people", "bill gates" and "the vaccine") switch insider-outsider label. even though the training data is highly biased in terms of the identities of the insiders/outsiders, the pretrained language model used in our classifier allows np2io to learn to infer using the context phrases and not by memorizing the labels. outsiders, strategies for dealing with outsiders and their attendant threats and, in the case of conspiracy theories, causal chains of events that support that theory. by itself, this unsupervised platform does not "understand" the different narrative parts. since the submodules are not trained to look for specific semantic abstractions inherent in conspiracy theories, the platform cannot automatically generate a semantically tagged narrative for downstream nlp tasks. it cannot, for example, generate a list across narratives of the various outside threats and attendant inside strategies being recommended on a social media forum, nor can it address why these threats and strategies are being discussed. ## the novel insider vs. outsider classification problem as a fundamental first step bringing in supervised information to enable automated narrative structure discovery, we introduce the insider-outsider classification task: to classify the noun phrases in a post as insider, outsider or neither. a working conceptualization of what we consider insiders and outsiders is provided in the following insets. as with most nlp tasks, we do not provide formal definitions of and rules to determine these groups. instead we let a deep learning model learn the representations needed to capture these notions computationally by training on data annotated with human-generated labels. ## our framework and related work recent nlp efforts have examined the effectiveness of using pretrained language models (lm) such as bert, distilbert, roberta, and xlm to address downstream classification tasks through fine-tuning @xcite @xcite . pretraining establishes the contextual dependencies of language prior to addressing a more specialized task, enabling rapid and efficient transfer learning. a crucial benefit of pretraining is that, in comparison to training a model from scratch, fewer labeled samples are necessary. by fine-tuning a pretrained lm, one can subsequently achieve competitive or better performance on an nlp task. as discussed in section 2, since our model is required to be contextual and transitive, both of which are qualities that rely on the context embedded in language, we utilize a similar architecture. in recent work involving span-based classification tasks, token-classification heads have proven to be very useful for tasks such as, parts-of-speech (pos) tagging, named entity recognition (ner) and variations of sentiment analysis (sa) @xcite @xcite . since the insider-outsider classification task is also set up as a noun phrase labeling task, our architecture uses a similar token-classification head on top of the pretrained lm backbone. current sa datasets' definitions of positive negative and neutral sentiments can be thought of as a "particularized" form of the insider-outsider classification task. for example, among the popular datasets used for sa, rotten tomatoes, yelp reviews @xcite and others @xcite implicitly associate a sentiment's origin to the post's author (source) (a single insider) and its intended target to a movie or restaurant (a single outsider if the sentiment is negative or an insider if positive). the post itself generally contains information about the target and particular aspects that the insider found necessary to highlight. in more recent sa work, such as aspect-based sentiment analysis (absa) @xcite @xcite , researchers have developed models to extract sentiments -positive, negative, neutral -associated with particular aspects of a target entity. one of the subtasks of absa, aspect-level sentiment classification (alsc), has a form that is particularly close to the insider-outsider classification. interpreted in the context of our task, the author of the post is an insider although now there can potentially be multiple targets or "aspects" that need to be classified as insiders and outsiders. still, the constructed tasks in absa appear to not align well with the goal of insider-outsider classification: 1) datasets are not transitive: individual posts appear to have only one agent that needs classification, or a set of agents, each with their own separate sets of descriptors; 2) the alsc data is often at the sentence-level as opposed to post-level, limiting the context-space for inference. despite these obvious differences, we quantitatively verify our intuitions in section 7.1, and show that absa models do not generalize to our dataset. closely related to absa is stance classification (sc) (also known as stance detection / identification), the task of identifying the stance of the text author (in favor of, against or neutral) toward a target (an entity, concept, event, idea, opinion, claim, topic, etc.) @xcite @xcite . unlike absa, the target in sc does not need to be embedded as a span within the context. for example, a perfect sc model given an input for classification of context: this house would abolish the monarchy. and target: hereditary succession, would predict the negative label @xcite . while sc appears to re-quire a higher level of abstraction and, as a result, a model of higher complexity and better generalization power than those typically used for absa, current implementations of sc are limited by the finite set of queried targets; in other words, sc models currently do not generalize to unseen abstract targets. yet, in real-time social media, potential targets and agents exhibit a continuous process of emergence, combination and dissipation. we seek to classify these shifting targets using the transitive property of language, and would like the language to provide clues about the class of one span relative to another. ultimately, while sc models are a valuable step in the direction of better semantic understanding, they are ill-suited to our task. parallel to this work in sa, there are complementary efforts in consensus threat detection on social media @xcite @xcite , a task that broadly attempts to classify longer segments of text -such as comments on youtube or tweets on twitteras more general "threats". the nuanced instruction to the labelers of the data is to identify whether the author of the post is an outsider from the labeler's perspective as an insider. once again, we observe that this task aligns with the insider-outsider paradigm, but does not exhaust it, and the underlying models cannot accomplish our task. the sets of insiders and outsiders comprise a higher-order belief system that cannot be adequately captured with the current working definitions of sentiment nor the currently available datasets. this problem presents a primary motivation for creating a new dataset. for example, the post: "microchips are telling the government where we are", does not directly feature a form of prototypical sentiment associated with "microchips", "the government" and "we", yet clearly insinuates an invasion on our right to privacy making clear the insiders ("we") and outsiders ("microchips", "the government") in the post. ## data collection to construct our novel dataset -conspiracy theory-5000 (ct5k) -we designed crawlers to extract a corpus of social media posts generated by the underlying narrative framework of vaccine hesitancy (details of the crawlers are documented in appendix a.1). vaccine hesitancy is a remarkably resilient belief fueled by conspiracy theories that overlaps with multiple other narratives including ones addressing "depopulation", "government overreach and the deep state", "limits on freedom of choice" and "satanism". the belief's evolution on social media has already enabled researchers to take the first steps in modeling critical parts of the underlying generative models that drive antivaccination conversations on the internet @xcite . moreover, vaccine hesitancy is especially relevant in the context of the ongoing covid-19 pandemic (burki, 2020). on the crawled corpus, we extract the nounchunks from each post using spacy's noun chunk extraction module and dependency parsers @xcite . a noun chunk is a subtree of the dependency parse tree, the headword of which is a noun. the result is a set of post-phrase pairs, (p, n), where p is a post and n is one of the noun phrases extracted from the post. amazon mechanical turk (amt) (see appendix a.2 for labeler instructions) was used to label the post-phrase pairs. for each pair, the labeler was asked, given the context, whether the writer of the post p perceives the noun phrase n to be an insider, outsider or neither (n/a). the labeler then provides a label @xmath0, where @xmath1, outsider, n/a} (hence |c| = 3). the triplets of post-phrase pairs along with their labels form the dataset @xmath2, n i ), c i |d| @xmath3. note that a single post can appear in multiple triplets, because multiple different noun phrases can be extracted and labeled from a single post. the overall class distribution and a few conditional class distributions across the labeled samples for several particular noun phrases are provided in figure 5 in the appendix b. manual inspection of the labeled samples ((p, n), c) suggests that the quality of the dataset is good (< 10% misclassified by random sampling). the now-labeled ct5k dataset @xcite 1 (|d| = 5000 samples) is split into training (90%), and 10% testing sets. 10% of the training set is held out for validation. the final training set is 20-fold augmented by bert-driven multi-token insertion @xcite . ## methodology and pipeline the noun-phrase-to-insider-outsider (np2io) model 2 adopts a token classification architecture comprising a bert-like pre-trained backbone and a softmax classifier on top of the backbone. token-1 see: data and model checkpoints 2 code repository: np2io level labels are induced from the span-level labels for the fine-tuning over ct5k, and the span-level labeling of noun phrases is done through majority vote during inference. ## baseline models in this section, we list baselines that we compare to our model's performance ordered by increasing parameter complexity. ## results and evaluation comparison of np2io to baselines is provided in ## concluding remarks we presented a challenging insider-outsider classification task, a novel framework necessary for addressing burgeoning misinformation and the proliferation of threat narratives on social media. we compiled a labeled ct5k dataset of conspiracytheoretic posts from multiple social media platforms and presented a competitive np2io model that outperforms non-trivial baselines. we have demonstrated that np2io is contextual and transitive via its zero-shot performance, adversarial studies and qualitative studies. we have also shown that the ct5k dataset consists of underlying information that is different from existing absa datasets. given np2io's ability to identify insiders and outsiders in a text segment, we can extend the inference engine to an entire set of interrelated samples in order to extract, visualize and interpret the underlying narrative (see figure 3 ). this marks a first and significant step in teasing out narratives from fragmentary social media records, with many of its essential semantic parts -such as, insider/outsider -tagged in an automated fashion. as extensive evaluations of the np2io model show, our engine has learned the causal phrases used to designate the labels. we believe an immediate future work can identify such causal phrases, yet another step toward semantic understanding of the parts of a narrative. broadly, work similar to this promises to expedite the development of models that rely on a computational foundation of structured information, and that are better at explaining causal chains of inference, a particularly important feature in the tackling of misinforma-figure 3 : an actor-actant subnarrative network constructed from social media posts: selected posts from anti-vaccination forums such as qresearch on 4chan were decomposed into relationship tuples using a state-ofthe-art relationship extraction pipeline from previous work @xcite and these relationships are overlayed with the inferences from np2io. this results in a network where the nodes are the noun phrases and the edges are the verb phrases, with each edge representing an extracted relationship from a post. in this network, a connected component emerged capturing a major sub-theory in vaccine hesitancy. this highlights np2io's ability at inferring both the threat-centric orientation of the narrative space as well as the negotiation dynamics in play, thereby providing qualitative insight into how np2io may be used in future work to extract large-scale relationship networks that are interpretable. the green boxes highlight the noun phrases that have contradictory membership in the insiders and the outsiders classes as their affiliations are deliberated.