Interactively Uncovering Latent Arguments in Social Media Platforms: A Case Study on the Covid-19 Vaccine Debate (2022)

Abstract: Automated methods for analyzing public opinion have grown in popularity with the proliferation of social media. While supervised methods can be very good at classifying text, the dynamic nature of social media discourse results in a moving target for supervised learning. Meanwhile, traditional unsupervised techniques for extracting themes from textual repositories, such as topic models, can result in incorrect outputs that are unusable to domain experts. For this reason, a non-trivial amount of research on social media discourse still relies on manual coding techniques. In this paper, we present an interactive, humans-in-the-loop framework that strikes a balance between unsupervised techniques and manual coding for extracting latent arguments from social media discussions. We use the COVID-19 vaccination debate as a case study, and show that our methodology can be used to obtain a more accurate, interpretable set of arguments when compared to traditional topic models. We do this at a relatively low manual cost, as 3 experts take approximately 2 hours to code close to 100k tweets.

## introduction public opinion plays an important role in the making of policy in pluralistic and democratic societies, as it allows the will of citizens to be heard and accounted for @xcite . as social media has become one of the main outlets for political and civic engagement @xcite , there is a growing body of work focused on automatically analyzing public opinion on social media. the applications studied include identifying the sentiment towards specific governmental measures @xcite , detecting and analyzing morally charged statements about current events @xcite , exploring how ordinary citizens frame political issues @xcite , and contrasting the stances expressed in social media with public opinion surveys @xcite . in all of these cases, the variables of interest are well defined, and substantial efforts are dedicated to creating manually annotated resources. in other words, we know which issues, governmental measures, or frames are of interest, and the problems can be framed as supervised learning tasks. in this paper, we take a step back and tackle the problem of defining the space of relevant variables to analyze public opinion online. given a widely debated topic, we put the focus on uncovering repeating arguments surrounding discussions on twitter. uncovering general themes from collections of unstructured textual resources is a common goal for researchers and practitioners across various disciplines. unsupervised techniques, like topic models and clustering methods, have been the de-facto nlp solution to this problem @xcite @xcite . while widely used, these methods often produce topics that are clearly incorrect to domain experts @xcite . for this reason, and despite the popularity of contemporary nlp techniques, a non-trivial number of recent studies on social media discourse rely on manual, qualitative coding methods instead @xcite @xcite . in this work, we strike a balance between unsupervised nlp techniques and manual coding by adopting a humans-in-the-loop approach. we use the twitter debate surrounding the covid-19 vaccine as a case study, and present an interactive framework to discover and define the space of arguments frequently cited as reasons to refuse or accept the vaccine. our framework is designed to address two main challenges: 1) given a large collection of tweets, how can experts effectively explore the data and identify a set of repeating arguments, and 2) given a known space of high-level arguments, how can experts create and refine a representation that improves the mapping from tweets to arguments. for example, in fig. 1a we can observe an initial clustering of arguments provided by the system. our initial goal is to obtain a resulting set of named arguments with high-quality examples as identified by the experts. similarly, in fig. 1b we can observe two overlapping arguments. our next goal is to source explanations from the experts that result in a better mapping from tweets to arguments. this can be achieved by expanding and refining the theme representation according to the explanations provided. the expected result is a comprehensive set of high-level arguments that explain the discussion about the covid-19 vaccine, and a partial mapping from tweets to their most likely argument. to tackle our goals, we design and implement a simple protocol that allows groups of experts to work together towards this goal, and introduce an interactive tool equipped with operations to facilitate the discovery and refinement of arguments in large language resources. our work is related to interactive systems that leverage clustering techniques to help users discover relevant topics @xcite , systems that exploit visualization techniques to label data interactively @xcite @xcite , and human-in-the-loop topic modeling approaches that let users refine discovered topics @xcite @xcite . the main differences between these systems and our work are that: 1) we do not assume a known space of rele-vant labels, 2) we let experts drive and influence the topic discovery procedure in addition to supporting the exploration, and 3) we support open-ended feedback from experts in natural language. our experiments show that our framework can be used to uncover a set of arguments that cover a large portion of the discussion about the covid-19 vaccine on twitter, and that the resulting mapping from tweets to argument is fairly accurate with respect to human judgements. additionally, we use the dataset of tweets about the covid-19 vaccine released by @xcite , which is annotated for vaccination stance and morality frames, and show that the high-level arguments obtained using our methodology have higher correlations with vaccination stance and moral sentiments than topics obtained using traditional topic models. ## interactive framework we propose a simple protocol that combines nlp techniques, interactive interfaces and qualitative methods to assist experts in characterizing large tweet repositories about the covid-19 vaccine. our protocol takes a large repository of tweets and automatically proposes an initial partition of the data, such that tweets that are thematically similar are clustered together. we provide experts with an interactive interface equipped with a set of exploratory operations that allows them to evaluate the quality of the discovered clusters, as well as to further explore and partition the space by inspecting individual examples, finding similar tweets, and using open text queries. as they interact with the data through interface, a group of experts work together following an inductive thematic analysis approach to identify and code the patterns that emerge within the partitions @xcite . next, they group the identified patterns into general arguments, and instantiate them using the interface. although intuitively we could expect a single cluster to result in a single argument, note that this is not enforced. experts maintain full freedom as to how many arguments they instantiate, if any. once an argument is created, experts are provided with a set of operations to explain the argument using natural language, select good example tweets, or write down additional examples. at any point during the process, experts can toggle a procedure that assigns tweets to arguments. ## case study as a case study, we look at tweets written about the covid-19 vaccine on twitter. we collected a corpus of 85,000 tweets that mentioned the vaccine. to avoid repetitions, we filter out all retweets ahead of time. the collected tweets are uniformly distributed between january and october, 2021. all tweets in our corpus are written in english, and were posted by users located in the united states. our main goal is to use the framework introduced in sec. 2 to identify repeating themes in this corpus, and construct a set of high-level arguments that are frequently used to justify stances on the vaccine. for our study, we recruited six experts in natural language processing and computational social science, four male and two female, within the ages of 25 and 45. the experts included graduate students, postdoctoral researchers and faculty. to evaluate the different components of our framework, we performed a two-stage analysis. in the first stage, we simplify the problem and assume that we have an initial, known set of high-level arguments, and let three of the experts focus on the challenge of interactively refining the arguments and grounding them in the large twitter corpus. in the second stage, we remove this assumption and have the remaining three experts discover the space of relevant arguments from scratch. below, we present each of these scenarios in detail and perform both qualitative and quantitative evaluations to assess the outcome of the interaction. ## conclusions we presented an initial step towards an interactive, humans-in-the-loop framework for uncovering latent arguments in social media discourse. we implemented a simple protocol that allows groups of experts to work together efficiently to create a comprehensive code-book of high-level arguments, and developed a gui with a set of computational operations to streamline their coding process. we used the covid-19 vaccine debate as a case study, and showed that by applying subsequent runs of our methodology, experts can obtain a comprehensive set of arguments that account for a reasonable slice of the data without sacrificing performance. additionally, we showed that our resulting set of arguments is cleaner and more explainable than themes obtained with topic modeling approaches. a.2 quality assurance operations figure 5: listing arguments and examples figure 6: visualizing local explanations: word cloud example for the vaccine doesn't work (a) stance figure 8: visualizing local explanations: attribute distribution for the vaccine doesn't work. note that attributes can be predicted using external resources. in this case, we predicted stance using a classifier trained on hashtags, as described in (pacheco et al., 2022). figure 10: visualizing global explanations: coverage figure 11: visualizing global explanations: 2d t-sne a.3 intervention operations figure 13: marking instances as good figure 14: adding good examples a.4 full screenshots full screenshots of the page views of our gui can be seen in figures 15, 16, 17, 18, 19, 20, 21 and 22 a.5 stage 1: coded patterns and contributed phrases coded patterns for each argument can be observed in tab. 8. the resulting list of added and removed arguments, as well as their contributed phrases can be observed in tabs. 9 and 10.