(Mis)alignment Between Stance Expressed in Social Media Data and Public Opinion Surveys (2021)

Abstract: Stance detection, which aims to determine whether an individual is for or against a target concept, promises to uncover public opinion from large streams of social media data. Yet even human annotation of social media content does not always capture “stance” as measured by public opinion polls. We demonstrate this by directly comparing an individual’s self-reported stance to the stance inferred from their social media data. Leveraging a longitudinal public opinion survey with respondent Twitter handles, we conducted this comparison for 1,129 individuals across four salient targets. We find that recall is high for both “Pro’’ and “Anti’’ stance classifications but precision is variable in a number of cases. We identify three factors leading to the disconnect between text and author stance: temporal inconsistencies, differences in constructs, and measurement errors from both survey respondents and annotators. By presenting a framework for assessing the limitations of stance detection models, this work provides important insight into what stance detection truly measures.

## introduction while surveys have a long history of quantifying public opinion @xcite , deploying them longitudinally requires considerable resources and may produce measurements which lag behind rapidly evolving events. particularly during ongoing or developing situations, such as the covid-19 pandemic and u.s. presidential elections, accurately capturing both current and historical opinion may be crucial to crafting appropriate policy interventions or anticipating political outcomes. in contrast to surveys, stance detection algorithms, which identify whether a given text is in favor or against a target concept, can be easily deployed at scale on social media data @xcite , positioning them as a potential supplement or replacement for traditional public opinion surveys @xcite . however, stance detection methods have rarely been evaluated in comparison to the public opinion data they promise to replace. instead, stance detection methods have almost exclusively been evaluated through annotated social media data, often produced by crowdworkers. when disagreements in stance occur across annotators or tasks, existing stance detection work often chooses to either ignore the disagreements @xcite @xcite , reconcile them with expert annotators @xcite , and/or account for them by modeling the annotation @xcite . this leaves two critical, largely open questions for stance detection. first, when do stance annotators agree (rq1)? annotating stance is a subjective process in which annotators use various signals to come to a conclusion @xcite . as has been observed in other areas of nlp, embracing this subjectivity and better understanding where disagreements arise can help us improve how we both construct and evaluate our models @xcite . second, how well do human annotations of the stance of individual users map to other means of inferring attitudes, and in particular, to survey responses (rq2)? if human coding of text is consistent with survey responses, then algorithms built on those annotations may plausibly replace survey methods. however, if this mapping is inconsistent, researchers may need to consider how stance detection can supplement survey methods, rather than replace them. here then, we aim to understand the degree to which human annotations of stance expressed in social media data are consistent across annotators and with public opinion polling. in cases where these measurements disagree, we deconstruct the likely causes of those disagreements. to do so, we leverage data from a longitudinal public opinion survey of americans conducted in 2020 and early 2021, where respondents voluntarily provided researchers with their twitter handle. we focus on survey responses to four topics salient in 2020: the u.s. presidential election, lockdowns in response to the pandemic, the use of face masks, and covid-19 vaccines. we then sample and annotate tweets from survey respondents on these topics. this allows us to directly compare an individual's stance as perceived by expert annotators and stance as self-reported on the survey. with respect to inter-coder alignment (rq1), we find that annotators vary significantly in their level of agreement based on the target of the stance detection task and how confident they are in their annotations. disagreements stem from two major sources: stance targets being too complex to reduce to a traditional three point-scale, and the existence of contradictory signals within the information provided for annotation. the alignment between annotations and the survey data (rq2) is impacted by the same factors as inter-annotator disagreement. however, these measures are also affected by temporal differences in when the surveys were taken and tweets were sent, and by differences in what individuals are willing to express on surveys as compared to twitter. all together, our work suggests that neither stance detection methods nor surveys should be seen as the "best" measurement tool, but rather as complementary constructs in the study of public opinion. as we discuss, this points to new directions for both stance detection methodology and the broader practice of connecting survey and so-cial media data. while, due to stringent privacy restrictions on the survey data used, we cannot release survey data, anonymized annotation data that can be used to replicate figures in the present work are included in the supplemental materials. ## data and methods in this section, we provide an overview of the survey and twitter data used, and how we constructed measures of stance for four different stance targets: donald trump, covid-related lockdowns, face masks, and covid-19 vaccines. we stress that these two datasets are linked, in that 14% of survey respondents provided a twitter handle, and so we can connect a subset of our survey respondents to the tweets that they sent. however, for clarity, we describe them separately below. ## discussion and conclusion we find important indicators of stance interpretability and how it compares to survey measures. with respect to when annotators agree with each other (rq1), agreement was very high when at least one annotator was very confident, i.e. when at least one believed stance was explicit. when both annotators were somewhat or not at all confident, i.e. when both felt they had to infer stance from indirect statements, agreement between annotators was moderate, low, or even worse than chance. with respect to how well annotations agreed with survey responses (rq2), the later respondents took the survey after a target was added to the survey, the more likely the respondent was to provide a non-neutral stance. if respondents did provide a non-neutral stance, annotations often matched with the survey data. there were, however, two exceptions to this. first, annotators rarely agreed with surveys for lockdowns, because they struggled to agree with each other. second, annotators also frequently labeled anti-mask and anti-vaccine survey stances as pro-mask or pro-vaccine, respectively, and labeled pro-trump survey responses as anti-trump. we have presented two possible reasons for these latter disagreements between survey and annotation: opinion change and different social norms on surveys as opposed to twitter. our findings are not without limitation. our operationalization of survey stance was one of many possible strategies. while the consistency of our main findings across four targets suggests that our overarching points are consistent with the data, choices we made here might nonetheless impact specific findings. further, we rely on respondents providing accurate handles for themselves, another potential source of error. we performed cleaning to try to ensure this, but our methods likely were not infallible. with these limitations in mind, our findings have several implications. first, one oft-stated claim in support of stance detection for social media is that, relative to surveys, it can better capture opinion dynamics. our work challenges that assumption at the individual level, because we find that individuals seem to tweet only when their minds are made up, whereas surveys can capture responses as individuals are making up their minds. in any case, many stance detection models leverage fea-tures about an individual that are relevant to past behaviors, making implicit assumptions about the static nature of individual stance. for example, a consistent feature in supervised models is who an individual follows @xcite , and more recent work has focused on historical retweeting patterns @xcite . even if stance itself were static, we find reason to be wary of assuming the meaning of these features are static as well. an example is an individual who retweeted a pro-mask tweet from rick scott, a u.s. senator for florida, early in the pandemic, which might mean something very different than a retweet of rick scott only months later as he shifted to a blend of pro-and anti-mask rhetoric. 4 second, and related, is that twitter may thus counter-intuitively be a bad place to look for opinion change at the individual level. from our analysis, we hypothesize that this is because the social norms on twitter disincentive individuals from publicly posting the kinds of internal debates they have on surveys that lead them to express more neutral attitudes. instead, twitter may be a better place to study public opinion at the "meso-level" (berinsky, 2017), studying how collective shifts in opinions from people who have already made up their minds change. third, our results suggest that there seem to be some targets on which stance detection is bound to be unreliable. our work suggests the main factor driving this is the complexity of the target in terms of the number of relevant policy aspects. lockdowns present a useful example for future work. because they pervaded so much of daily life, stance on lockdowns was complex and nuanced, focusing on everything from homeschooling to universal basic income. the number of survey questions we combined to compute a lockdown stance was itself indicative of this point. indeed, the way we aligned these survey questions to stance may have impacted our results. however, since prior stance detection work has not compared its output with individual-level opinion polls, there are no established guidelines for doing so. a potential avenue for future work, then, might seek to address aspects of stance in the way we now do for, e.g., sentiment @xcite and opinion @xcite mining. at an overarching level, our results suggest that stance detection applied to social media data does not capture the same thing that is captured by public opinion polling. this should not necessarily come as a surprise. stance detection methods are evaluated by their ability to determine individuals' attitudes as they are expressed on social media and perceived by an annotator. in contrast, polling data captures attitudes solicited by a researcher and selfperceived by individuals. our observations here suggest potential merit in leveraging, rather than setting aside, these differences.