Our kind of people? Detecting populist references in political debates (2023)

Abstract: This paper investigates the identification of populist rhetoric in text and presents a novel cross-lingual dataset for this task. Our work is based on the definition of populism as a “communication style of political actors that refers to the people” but also includes anti-elitism as another core feature of populism. Accordingly, we annotate references to The People and The Elite in German and English parliamentary debates with a hierarchical scheme. The paper describes our dataset and annotation procedure and reports inter-annotator agreement for this task. Next, we compare and evaluate different transformer-based model architectures on a German dataset and report results for zero-shot learning on a smaller English dataset. We then show that semi-supervised tri-training can improve results in the cross-lingual setting. Our dataset can be used to investigate how political actors talk about The Elite and The People and to study how populist rhetoric is used as a strategic device.

## introduction the rise of populism in europe and throughout the world has been noted not only in politics and the media but also has been the subject of many studies in political science and related areas (see, among others, @xcite ). the concept of populism, however, is complex and vague and eludes a strict definition. so far, only limited agreement exists on the exact properties of the construct, despite numerous efforts to provide a clear definition. in the literature, populism has been described as an ideology @xcite , a rhetoric @xcite or style @xcite , as a political strategy @xcite @xcite and as a discourse @xcite , amongst others (see @xcite for a short overview). the oxford handbook on @xcite groups existing work into three dominant approaches to analyzing populism, i.e., (i) the ideational approach of @xcite , (ii) the socio-cultural approach @xcite , and (iii) the political-strategic approach @xcite , each one capturing a different view on populism. nevertheless, most studies agree that anti-elitism and people-centrism are amongst the core dimensions of populist rhetoric, and the two dimensions are therefore included as features in most survey tools used to measure the degree of populism of political parties and actors @xcite @xcite . one major drawback of surveys, however, is that they only provide us with one score for each party or actor and can not be used to study how populist rhetoric is used as a strategic tool in different contextual settings. as a result, more and more efforts have been made recently to measure populist and anti-elitist attitudes directly from text @xcite @xcite @xcite @xcite . this has the advantage of providing us with more fine-grained and contextdependent measures that enable us to investigate when and how anti-elitist rhetoric is used as a strategic tool in party competition @xcite . in addition, it has been suggested that populist rhetoric targeting political elites might function "as a form of ethnoracial dog-whistle politics" @xcite . evidence for this claim comes from the frequent co-occurrence of right-wing populism with nativist messages, as shown in example 1.1 below, taken from a parliamentary speech of a far-right politician in the german bundestag. ## mope: annotating mentions of the people and the elite we now present mope, our new data set with annotated mentions of the people and the elite. the people versus the elite. according to @xcite , the difference between the two camps in populist rhetoric is not based on issues of class or nationality, but rather on morality. the people are an artificial construct of a (non-existing) homogeneous community whose defining criteria are self-ascribed and depend on the specific ideology that serves as the carrier for the thin-centered ideology, i.e., populism (see §2.1). the elite, on the other hand, can be seen as the anti-thesis of the the people and also obtains its defining features based on the situational context. to operationalize the two concepts, we use a hierarchical schema where we encode instances of the two classes on the first level ( german bundestag data. we extracted a dataset of german parliamentary debates for the 19th legislative term (2017-2021), controlled for topic and party membership of the speakers. 2 the time frame was selected because of its relevance for the rise and consolidation of populist rhetoric in german politics. our data set includes 267 speeches by 196 different speakers from 6 german parties ( annotation process. the data was double annotated by two student assistants with background in political/social science. during the annotation process, we had weekly meetings to discuss ambiguous cases. the final version was adjudicated by one of the authors (a linguist by training), who also corrected inconsistent span annotations: it includes 9,297 annotated mentions (german subcorpus). in our experiments, we ignore all mentions where the speakers refer to themselves (label epown) using the pronouns i/me, since this label can be assigned based on a simple string match. this results in a set of 7,422 mentions with 22,479 annotated tokens that we divide into training, dev and test set (see appendix b, the english data set includes 29,584 tokens with 1,423 annotated mentions (1,074 w/o epown) and 3,567 annotated tokens (3,218 w/o epown). ## experiments we use our data set from §3 to benchmark the task of predicting mentions of the elite and the people from text sentences. our task can be decomposed into two separate sub-tasks: (i) mention detection (md) and (ii) mention classification (mc). we present experiments where we compare different transformer-based model architectures @xcite for those tasks. specifically, we compare (i) a pipeline approach (md-→mc) with (ii) an end-to-end token classification model (e2e-tok) and (iii) semi-supervised tritraining (tri) @xcite . ## measuring thin populism from text we are now able to investigate jagers and walgrave (2007)'s concept of thin populism by looking at how often political actors refer to different subsets of the people. for that, we use our three monolingual classifiers described in §4 and predict labels for all debates from the german bundestag from the 19th legislative term (2017-2021) (> 16 million tokens). we take the majority vote of the three classifiers to determine the final predictions. figure 2 shows the distribution of the aggregated counts for all references to the people for each party. 5 we can now validate how well our operationalization of thin populism in text correlates with expert ratings. for that, we compute spearman's rank correlation between the normalized counts for each party and the party's score for people-centrism in the populism and political parties expert survey (poppa) (meijers and zaslove, 2021b) (also see however, when looking at the fine-grained predictions for different subgroups of the people 5 we excluded the csu from the analysis. while the party is forming a joint parliamentary group with the cdu in the bundestag, it is only running for election in a single german province, bavaria. this results in a conflict between the party's "bavaria first!" policy on the province level and the need to accommodate their sister party's policies on the federal level @xcite . we, therefore, expect that the governing faction is not representative of the party as a whole. (level 3, figure 3 ), we also notice interesting differences. for example, both populist parties use a higher amount of references to people by func-tion than the mainstream parties. at the same time, only the far-right afd shows excessive use of people by nation, often as a dog-whistle to send the message that some people are not "our kind of people". 6 overall, our approach of predicting references to the people is able to successfully identify populist rhetoric in large amounts of text and agrees well with expert ratings. however, our results also highlight the importance of a more fine-grained operationalization of thin populism that distinguishes between different subgroups of the people. ## conclusions in this paper, we presented mope, a novel data set for detecting mentions of the people and the elite in political text. our data set includes more than 9,000 annotated mentions for german and an english benchmark set with around 1,600 mentions for cross-lingual transfer learning. we evaluated different transformer-based model architectures on our new data set and explored zero-shot cross-lingual transfer and cross-lingual tri-training. in future work, we will combine references to the elite with stance detection, which will allow us to model and quantify the different dimensions of populism separately, i.e., people-centrism and anti-elitism, thus enabling large-scale studies of populism from left-and right-wing political actors in different contextual settings.