Optimizing text representations to capture (dis)similarity between political parties (2022)

Abstract: Even though fine-tuned neural language models have been pivotal in enabling “deep” automatic text analysis, optimizing text representations for specific applications remains a crucial bottleneck. In this study, we look at this problem in the context of a task from computational social science, namely modeling pairwise similarities between political parties. Our research question is what level of structural information is necessary to create robust text representation, contrasting a strongly informed approach (which uses both claim span and claim category annotations) with approaches that forgo one or both types of annotation with document structure-based heuristics. Evaluating our models on the manifestos of German parties for the 2021 federal election. We find that heuristics that maximize within-party over between-party similarity along with a normalization step lead to reliable party similarity prediction, without the need for manual annotation.

## introduction a party manifesto, also known as electoral program, is a document in which parties express their views, intentions and motives for the next coming years. since this genre of text is written not just to inform, but to persuade potential voters that the parties compete for @xcite , it provides a strong basis to understand the position taken by parties according to various policies because of its direct access to the parties' opinions. political scientists study the contents of party manifestos, for instance, to investigate parties' similarity with respect to the several policies @xcite , to predict party coalitions @xcite , and to evaluate the extent to which the parties that they vote for actually corresponds to their own world view @xcite . to carry out systematic analyses of party relations while taking into account differences in style and level of detail, these analyses are increasingly grounded in two types of manual annotation about claims, statements that contain a position or a view towards an issue, that can be argued or demanded for @xcite : first, abstract claim categories @xcite are used to group together diverse forms and formulations of demands. second, annotation often includes the stance that parties take towards specific political claims to abstract away from the many ways to express support or rejection in language. in addition, these types of annotation offer a direct way to empirically ground party similarity in claims and link these to concrete textual statements. at the same time, such manual annotation is extremely expensive in terms of time and resources and has to be repeated for every country and every new election. in this paper, we investigate the extent to which this manual effort can be reduced given appropriate text representations. we build on the advances made in recent years in neural language models for text representations and present a series of finetuning designs based on manifesto texts to compute party similarities. our main hypothesis is that the proximity between groups can be more easily captured when the model receives adequate indication of the differences between groups (and their stances) and this can be done via fine-tuning for instance. this can be achieved by using signal that is freely available in the manifestos' document structure, such as groupings by party or topic. information of this type can serve as an alternative feedback for fine-tuning in order to create robust text representations for analysing party proximity. we ask three specific questions: (1) how to create robust representations for identifying the similarity between groups such as in the case of party relations? (2) what level of document structure is necessary for this purpose? (3) can computational methods capture the relation between parties in unstructured text? we empirically investigate these questions on electoral programs from the ger-man 2021 elections, comparing party similarities against a ground truth built from structured data. we find that our hypothesis is borne out: we can achieve competitive results in modelling the party proximity with textual data provided that the text representations are optimized to capture the differences across parties and normalized to fall in a certain distribution that is appropriate for computing text similarity. more surprisingly, we find that completely unstructured data reach higher correlations than more informed settings that consider exclusively claims and/or their policy domain. we make our code and data available for replicability. 1 paper structure. the paper is structured as follows. section 2 provides an overview of related work. section 3 describes the data we work with and our ground truth. section 4 presents our modeling approach. sections 5 and 6 discuss the experimental setup and our results. section 7 concludes. ## data before we describe the methods we will use, we describe our textual basis and the ground truth we will aim to approximate. ## methods we describe our method in three steps: (a) we define a set of informative text representations models; (b) we compute party similarities, parallel to section 3.2, on the basis of these text representations; (c) we post-process the data. ## results and discussion ## conclusion in this paper, we have investigated to what degree text representations can capture the proximity of parties and how to best fine-tune representations for this task. our results indicate that aspects that have been proposed as important for this type of analysis in political science, namely annotation of domains @xcite and claims @xcite , do not appear to matter greatly for this task -or at least, manual annotation can be replaced by nlp tools: we have recognized claims with a classifier @xcite and have proposed a weekly supervised method, "twin matching", to approximate domain-level similarity computation. indeed, one of our models that does not use any manual annotation is among the top contenders. of rather greater importance for party similarity prediction, according to our findings, is fine-tuning the text representations and postprocessing them. this is good news for computational political science: the judicious use of document structure appears able to help alleviate the effort of having domain experts annotate large corpora. the two main limitations of our current study relate to this outlook: (a) we only experimented with a single language and ground truth -future work should take into account multiple languages and time periods, with a potential long term goal of text-based models for party development @xcite ; (b) we only scratched the surface of cues available for fine-tuning. future work could, for example, take into account other aspects of parties such as ideological position @xcite , or reach beyond manifestos to include information from other types of party interactions @xcite . in addition to that, work on interpreting both the fine-tuned and vanilla sbert models would be interesting to better understand the predominant dimensions of the sentence representations in the political domain.