Multilingual estimation of political-party positioning: From label aggregation to long-input Transformers (2023)

Abstract: Scaling analysis is a technique in computational political science that assigns a political actor (e.g. politician or party) a score on a predefined scale based on a (typically long) body of text (e.g. a parliamentary speech or an election manifesto). For example, political scientists have often used the left–right scale to systematically analyse political landscapes of different countries. NLP methods for automatic scaling analysis can find broad application provided they (i) are able to deal with long texts and (ii) work robustly across domains and languages. In this work, we implement and compare two approaches to automatic scaling analysis of political-party manifestos: label aggregation, a pipeline strategy relying on annotations of individual statements from the manifestos, and long-input-Transformer-based models, which compute scaling values directly from raw text. We carry out the analysis of the Comparative Manifestos Project dataset across 41 countries and 27 languages and find that the task can be efficiently solved by state-of-the-art models, with label aggregation producing the best results.

## introduction a widely used tool in computational political science is the so-called 'scaling analysis': a set of methods for representing political platforms as numbers on a certain scale, such as leftright, authoritarian-libertarian, or conservativeprogressive @xcite @xcite @xcite . a wide variety of scales have been proposed in the literature, some based on political-theoretic considerations @xcite , others more data-driven @xcite @xcite . one well-established scoring scheme of this kind is the standard right-left scale, also known as the rile score @xcite . it was developed in the framework of the manifesto research on political representation project (marpor), formerly known as the comparative manifestos project (cmp), foot_0 which collects, annotates, and makes available a large collection of party platforms from different countries. the rile score is a deductive, first-principle-based method for describing party positions geared towards the widest possible applicability across time and countries @xcite . for this very reason, it is rather conservative and inflexible and has been repeatedly criticised (see, e.g., @xcite . despite this, it is widely used in computational political science for model validation @xcite , as a dependent variable in regression analyses @xcite , or as a basis for party-stance analysis @xcite . a major practical drawback of the rile score is the fact that it is computed based on the labels manually assigned by marpor annotators to all statements in party manifestos (see section 2 for details). this procedure is expensive and time consuming, which raises the question of whether we can adequately approximate the rile score using natural-language-processing methods, especially in a multilingual setting. this will make it possible to efficiently perform analyses of political texts that have not yet been covered by the marpor project due to timing constraints (e.g. manifestos from upcoming elections), accidental gaps (e.g., indonesia and the philippines are not part of the dataset, and coverage of many countries, such as south africa, is incomplete), or lack of resources (there are very few annotated manifestos from before 2000). this work is a first step in this direction. our contributions are the following: the paper is structured as follows: in § 2, we provide more information on the marpor annotations and on how the rile score is computed. the exact problem statement, different operationalization strategies, and the experimental setup are presented in § 3, while the results of the study are given in § 4. additional discussion is provided in § 5. section 6 surveys related work. section 7 concludes the paper and discusses directions for future research. ## marpor categories and political scales categories the annotations of the manifesto created in the framework of the comparative manifestos project follow the project codebook @xcite . each statement of a given manifesto is annotated with a category representing a specific policy domain (e.g. military or sustainability). these categories can be identified via their names and numbers (e.g., 103, anti-imperialism). 2 a key feature of marpor categories is that they are not stance-neutral. thus, category 201, freedom and human rights, or subtypes thereof, are assigned to 'favourable mentions of importance of personal freedom and civil rights' @xcite . some categories form binary oppositions (e.g. constitutionalism: positive vs. constitutionalism: negative), and some are purely onesided (e.g. freedom and democracy have positive loadings and do not have negative counterparts). as a result, it is possible to derive inferences about political stances of different parties from category counts alone. this provides a straightforward operationalization of the political-science notion of issue salience, which is commonly used to analyse political positioning @xcite the number of occurrences of a category correlates with how important it is for a party. in total, there are 143 different categories, with 56 major categories, 32 sub-categories of the major categories, 54 additional categories, and the residual category 0. foot_2 right-left scale a prominent way of analysing party positioning is the standard right-left scale, a.k.a. rile score @xcite . originally developed in the framework of the marpor project, it has been consistently used in its publications and remains a standard reference scale for party positioning, despite a number of proposals to improve or replace it, both using theorybased and data-driven approaches (cf. @xcite @xcite . ## results the main results of the experiments are summarised in tables 2 and 3 . sections 4.1 and 4.2 discuss the results while § 4.3 provides some detail about the strengths and weaknesses of the models. ## discussion our results show that multi-lingual automatic analysis of political-party positioning is at least partially feasible. it is possible to provide a high-level overview of the party system in a new country with a reasonable degree of precision, and even better results can be achieved with some amount of incountry data: the rile scores computed using our method demonstrate a remarkably high correlation with the gold scores. interestingly, the main obstacle to the success of our method seems to be not the language barrier, which is bridged well by either the off-the-shelf mt systems or the multilingual encoder, but the differences in the political culture across countries: the models struggle to correctly identify right-wing statements in the manifestos. in practical terms, using long-input transformers instead of sentence-level classifiers offers a way to greatly simplify the analysis and obviate the problems of subsentence identification in the input, as such models are able to make holistic judgements about long spans of text. in terms of performance, lits struggle on the task of directly estimating rile, compared to label-aggregation models, with the best model only approaching a reasonable level of performance. however, this must be taken with a grain of salt, since the label aggregation models have the advantage of gold-statement boundaries. furthermore, our binned-regression experiment shows that lits are promising candidates for coarse-grained party positioning analysis in terms of political 'camps'. for all models, the tails of the distribution remain hard to identify, with extreme categories rarely predicted correctly and centre left/centre right labels often mistaken for centrist. ## related work the work on computational analysis of political documents traditionally employs bag-of-words methods, such as those popularised by @xcite and @xcite . @xcite introduce distributional semantics in the leftright analysis by using multilingual word alignment in the embedding space and a graph-based scorepropagation algorithm. this approach is then built upon by @xcite . @xcite adapt the word2vec methodology to the analysis of parliamentary speeches in a single-language setting via the use of trained party vectors, whose dimension-ality they reduce using pca; they then interpret one of the resulting axes as the left-right scale. @xcite instead develop a methodology for identifying the political position of lawmakers on the progressive-to-moderate dimension with a bag-of-words-based topic-modelling approach. the use of contextualised embeddings for political analysis has not yet become mainstream. @xcite test a wide range of methods, from unigram statistics to bert-based classifiers, for assigning marpor labels to classify debate motions from the uk parliament. @xcite use several pre-trained single-language bert models for the task of political-statement classification in five languages. facing the same issues of label-frequency imbalance and rare labels, they mitigate them to some degree by using the hierarchical organisation of marpor labels; they do not try to compute rile scores. @xcite introduce sentence transformers @xcite into the problem space and fine-tune the embedding model itself in order to learn a politically informative distance measure between manifesto texts. @xcite further extend this method to analyse interparty differences with regard to major policy domains, such as law and order or sustainability and agriculture. more generally, our work falls into the domain of zero-shot classification with test data coming from a country or a time period not covered by the training data. the question of whether machine translation @xcite or multilingual encoders @xcite is better suited for cross-lingual transfer is still actively debated, and we explore both options. from another perspective, the task of identifying and characterising political positions from textual data abuts larger fields of stance detection and argument mining @xcite . ## conclusion in this paper, we have proposed the first series models that generalise the task of political-party positioning across countries and election cycles. we showed that the main challenge -predicting marpor labels across countries and election cycles with high accuracy -is, surprisingly, not a real barrier on the way to a highly precise multilingual scaling analysis. we experimented with the standard right-left scale (rile score), which is widely discussed in the political-science literature, and demonstrated that party manifestos can be effectively characterized in these terms using state-of-the-art multilingual modeling techniques applied to sentence-level classification with subsequent label aggregation and that even better results can be achieved via task-specific label clustering. we further experimented with replacing the label-aggregation approach with long-input transformers -both using regression and classification formulations -in order to obviate the task of identifying spans of statements from manifestos. these models demonstrate promising performance but still underperform the more traditional pipeline mimicking manual analysis. bridging the gap between long-input models and political analysis is an important avenue for future work, together with tackling other political dimensions and further widening the scope of the analysis.