Us vs. Them: A Dataset of Populist Attitudes, News Bias and Emotions (2021)

Abstract: Computational modelling of political discourse tasks has become an increasingly important area of research in the field of natural language processing. Populist rhetoric has risen across the political sphere in recent years; however, due to its complex nature, computational approaches to it have been scarce. In this paper, we present the new Us vs. Them dataset, consisting of 6861 Reddit comments annotated for populist attitudes and the first large-scale computational models of this phenomenon. We investigate the relationship between populist mindsets and social groups, as well as a range of emotions typically associated with these. We set a baseline for two tasks associated with populist attitudes and present a set of multi-task learning models that leverage and demonstrate the importance of emotion and group identification as auxiliary tasks.

## introduction political discourse is essential in shaping public opinion. the tasks related to modelling political rhetoric have thus been gaining interest in the natural language processing (nlp) community. many of them focused on automatically placing a piece of text on the left-to-right political spectrum. for instance, much research has been devoted to detecting bias in news sources @xcite and predicting the political affiliation of politicians @xcite and social media users, more generally @xcite @xcite . other works conducted a more fine-grained analysis, identifying the framing of political issues in news articles @xcite . recently, the field has also turned attention towards modelling the spread of political information in social media, such as detecting fake news or political perspectives @xcite @xcite . populism has taken the spotlight in political communication in recent years. various countries around the globe have experienced a surge of populist rhetoric @xcite in both the public and political space. despite this, approaches to computational modelling of populist discourse have so far been scarce. due to the flexible nature of populism, annotating populist rhetoric in text is challenging, and the existing research in this area has relied on small-scale analysis by experts @xcite . in this paper, we present a new dataset foot_0 of reddit comments annotated for populist attitudes and the first large-scale computational models of this phenomenon. we rely on research in social-and behavioural sciences (e.g., political science and social psychology) to operationalise a definition of populism and an annotation procedure that allows us to capture and generalise the crucial aspects of populist rhetoric at scale. in social sciences, populism is essentially described as a not fully developed political ideology and a series of background beliefs and techniques @xcite , traditionally centred around the us vs. them dichotomy. in one of the first attempts to fully define populism @xcite , it is described as a thin ideology around the distinction between 'the people', which includes the 'us', and 'the elites' describing the 'them', and with politics being a tool for 'the people' to achieve the common good or 'the popular will' @xcite . through different platforms, populism uses this rhetoric that revolves around social identity @xcite and the us vs. them argumentation @xcite . while right-wing populism tends to be characterised by fear, resentment, anger and hatred, left-wing populism is associated with shame and guilt @xcite . moreover, emotions have been shown to be crucial in shaping public opinion more generally @xcite @xcite . the design of our annotation scheme and the dataset are inspired by this research, particularly the link between populist rhetoric and both social identity and emotions. our dataset consists of comments posted on reddit that explicitly mention a social group. we collect the comments posted in response to news articles across the political spectrum. through crowd-sourcing, we annotate supportive, critical and discriminatory attitudes towards the group, as well as a range of emotions typically associated with populist attitudes. at the same time, given the relevance of news in the spread of such mindsets, we investigate the relationship between news bias and the us vs. them rhetoric. our data analysis reveals interesting interactions between populist attitudes, specific social groups and emotions. we also present a series of computational models, automatically identifying populist attitudes, based on roberta @xcite . we experiment in a multi-task learning framework, jointly modelling supportive vs. discriminatory attitudes towards a group, the identity of the group and emotions towards the group. we demonstrate that joint modelling of these phenomena leads to significant improvements in detection of populist attitudes. ## dataset creation data collection. by annotating reddit comments that refer to a social group, we monitored how online discussions target them and whether the text showed a positive or negative attitude towards that social group, ranging from support to discrimination. while this process did not ensure capturing the complexity behind the us vs. them rhetoric, we detected comments directed at certain groups (out-groups) and the attitude towards them within an online community (in-group). we restricted this to six specific groups that populist rhetoric has targeted as an out-group, immigrants, refugees, muslims, jews, liberals and conservatives. current research has shown these groups are common targets of populism in the us, uk and across europe @xcite . note that to annotate sufficient comments per group we limit the current work to six groups, which is by no means a complete list of targeted groups. we encourage future research to broaden the scope of groups covered. we chose to extract data from reddit, (1) due to its availability through the pushshift repository foot_1 @xcite and the google bigquery service, (2) its social identity dynamics (in-group vs. out-group) as close-nit communities created by sub-reddits, (3) its nature as a social news aggregation platform, and (4) that it has been shown to encourage toxic communication between users and hate speech towards social groups @xcite @xcite . to filter the data for annotation, we followed several steps. (1) we identified submissions in reddit which shared a news article from a news source listed at the allsides website foot_2 , (2) we extracted comments which are direct replies to the submission where both the news article title and the comment match any of the keywords for our groups. keywords were devised using online resources from the anti defamation league foot_3 as well as by consulting social scientists. the full list of keywords can be found in appendix a annotation procedure. to capture the us vs. them rhetoric, we asked: what kind of language does this comment contain towards group ?, where group corresponds to the specific social group that comment refers to. respondents had four options: discriminatory, critical, neutral or supportive. an extended description and an example as presented to annotators can be found in appendix a.2, and figure 5 . we asked anno-tators a second question to capture the emotions expressed towards the group in the same comment. we extended ekman's model of 6 basic emotions @xcite to a 12-emotions model, which includes a balanced set of positive and negative sentiments. specifically, we included emotions previously shown to be associated with populist attitudes @xcite . we also provided the annotators with a brief description of each emotion, inspired by the concept of core relational themes @xcite . the positive emotions are gratitude, happiness, hope, pride, relief and sympathy, and the negative emotions are anger, fear, contempt, sadness, disgust and guilt. detailed descriptions can be found in appendix a.2 along with an example 6. the annotation was conducted on amazon mechanical turk (mturk) and its framework can be accessed here.