On the Reliability and Validity of Detecting Approval of Political Actors in Tweets (2020)

Abstract: Social media sites like Twitter possess the potential to complement surveys that measure political opinions and, more specifically, political actors’ approval. However, new challenges related to the reliability and validity of social-media-based estimates arise. Various sentiment analysis and stance detection methods have been developed and used in previous research to measure users’ political opinions based on their content on social media. In this work, we attempt to gauge the efficacy of untargeted sentiment, targeted sentiment, and stance detection methods in labeling various political actors’ approval by benchmarking them across several datasets. We also contrast the performance of these pretrained methods that can be used in an off-the-shelf (OTS) manner against a set of models trained on minimal custom data. We find that OTS methods have low generalizability on unseen and familiar targets, while low-resource custom models are more robust. Our work sheds light on the strengths and limitations of existing methods proposed for understanding politicians’ approval from tweets.

## introduction measuring public opinion accurately and without systematic errors is as vital for a functioning democracy as it is for scholars to understand society. survey methodologists have developed techniques over several decades to precisely quantify public opinion. the american association for public opinion research (aapor) stated in their recent task force report that public opinion research is entering a new era, where digital traces would play an important role @xcite . increasingly, since the first steps were made by @xcite , numerous studies have assessed the efficacy of such traces, especially social media, in measuring public opinion as a complement to polls. the run for social media approaches is not surprising, as they promise a continuous public opinion estimate based on millions of data points. tweet untargeted sentiment targeted sentiment stance approval trump is the only candidate i fully support positive positive favor approval what makes me angry is the lying media brazenly attacking president trump. negative positive favor approval jeb bush is the only sane candidate in the republican lineup positive none against disapproval social-media-based metrics require new approaches, which bring forth new challenges @xcite . @xcite describe two primary sources of errors: representation errors, due to how results are inferred for the target population and measurement errors, due to how the target construct is measured. researchers have made substantial advances in understanding and adjusting for representation errors @xcite . yet, there is still a gap in knowledge about whether the lack of effectiveness of social media-based estimates is also due to measurement errors, i.e., the operationalization of the target construct -approval. while previous research has used external data such as polling results to (in)validate the efficacy of automated aggregate approval measures from social media, the fine-grained (mis)measurement of approval on a post level has yet to be studied. the building blocks for measuring approval with social media are usually the textual utterances by users. 1 related work predominantly focuses on the largest publicly available social media platform, twitter, and employs methods ranging from sentiment lexicons @xcite to machine learning approaches @xcite for analyzing approval in individual tweets. several natural language processing (nlp) approaches have been proposed to, or can be amended to, measure approval. they can be segmented into three broad classes: untargeted sentiment detection, targeted sentiment detection, and stance detection (see therefore, we investigate the following use case: measuring how respondents or users feel towards a 1 these can further be aggregated per user @xcite ), but we focus on the much more common practice of measuring post-level opinions. in the section, we describe widely used methods that have been applied to mine public opinion on twitter, particularly approval of political actors. evaluating all pertinent methods and their varying implementations is beyond the scope of this work, therefore we choose popular approaches or those whose implementations are widely available. we describe the three above-mentioned categories of approaches (summarized in ## use case scenarios we now describe the two scenarios we explore as realistic options faced by a css researcher aiming to measure approval towards political actors on twitter with their own dataset and/or targets. ## experiments previous research established the validity of social media measures through correlations with external data sources like polls and surveys @xcite @xcite . we argue that this entangles different types of errors, such as the lack of demographic match between polls and social media users and the effect of the platform's affordances on textual expressions. by focusing on a controlled dataset of human-annotated approval at a tweet level, we can rule out confounding factors to a higher degree. furthermore, as we see in ## results we now describe our findings from the the experiments described in the previous subsection. we compare the performance of methods across different targets in ## discussion one of the goals of language technology, including nlp methods, is the ability to (re-)use them. keeping in mind this vision, we investigate how an important construct in css, political approval, can be operationalized using existing nlp techniques, either through off-the-shelf sentiment and stance detection methods or through custom domain-specific methods. by comparing the performance of twelve methods over five datasets with approval towards seven targets, we find that targeted ots methods do not perform well across targets or datasets that span over different time periods and have been collected using different collection strategies. concretely, (i) targeted ots methods do not generalize beyond the targets they were trained on. they are as good as or even worse than general-purpose lexicons in this case; (ii) even for familiar targets, targeted methods, especially stance detection, have high fluctuations and perform worse than sentiment lexicons for certain datasets; (iii) finally, stance methods do not have a clear advantage over targeted sentiment in understanding approval due to the latter's low performance on indirect stance. while researchers interested in measuring approval should use targeted constructs like stance or targeted sentiment instead of overall sentiment to avoid conceptual confusion, current targeted methods need to be improved before they can be used in an off-the-shelf manner. since ots targeted methods do not perform well for unknown targets, authors of papers on stance detection and target-dependent sentiment analysis should clarify if their method works only for certain targets (target-specific) or can be used to measure stance towards any unseen target (general-purpose), i.e., clarify the borders of their method's applicability. the high performance of sentiment lexicons, especially for unseen targets ( limitations. this work does not capture all methods that have been proposed for assessing political approval but focuses on those that have been popular in the past or are exemplary for different types of methods (untargeted sentiment, targeted sentiment, and stance). second, we only consider approval towards named entities, which we find is already a difficult task, especially for indirect stances. in the future, we hope to explore abstract topics like 'immigration' where differentiating between direct and indirect stance is non-trivial and ensemble models that combine the strengths of multiple methods.