Swimming with the Tide? Positional Claim Detection across Political Text Types (2020)

Abstract: Manifestos are official documents of political parties, providing a comprehensive topical overview of the electoral programs. Voters, however, seldom read them and often prefer other channels, such as newspaper articles, to understand the party positions on various policy issues. The natural question to ask is how compatible these two formats (manifesto and newspaper reports) are in their representation of party positioning. We address this question with an approach that combines political science (manual annotation and analysis) and natural language processing (supervised claim identification) in a cross-text type setting: we train a classifier on annotated newspaper data and test its performance on manifestos. Our findings show a) strong performance for supervised classification even across text types and b) a substantive overlap between the two formats in terms of party positioning, with differences regarding the salience of specific issues.

## introduction electoral programs described in party manifestos are official documents of political parties. not only do they capture their positions on a broad variety of policy issues, but they are also authoritative for the entire party. in this sense, they are fundamentally different from statements issued only by factions or solitary members of a party @xcite . ordinary citizens, however, rarely read the manifestos @xcite . arguably, an informed voter is more likely to be exposed to electoral programs indirectly through another medium, namely the newspaper. news articles present the reader with the political claims put forward by different parties and often directly contrast them. more specifically, we adopt the definition of claims as any form of politically motivated demand or action (both verbal and non-verbal) of deliberate actors @xcite . information about claims is routinely used by readers to infer the parties' standpoints. this places parties in a delicate situation: they largely rely on newspapers and other media to convey their claims to the voter to gain their approval in elections @xcite . recent advances in the application of nlp methods to political reporting can reliably extract such claims with models trained on manually annotated claims in newspapers @xcite . from a nlp perspective, this gives rise to the following question which we will address in this paper: can we transfer existing models for claim identification to party manifestos without substantial loss of performance? if so, this would enable political scientists to address substantive questions such as: to what extent do manifestos of specific parties and the positions of their members as reported in newspaper articles overlap? how do differences articulate themselves in practice? one way to address these questions is to directly compare the agreement between claims from the two formats foot_0 . figure 1 (left panel) shows four sentences containing political claims on the pol-icy issue of migration. while the first sentence is translated from the party manifesto of the german left-wing party linke (left) from the federal election 2013, the last one is from the 2017 manifesto of the conservative christian democratic union (cdu). both sentences in the middle are statements from the german minister of the interior (cdu) published in newspaper articles in 2015. despite the discrepancy in time and genre, a) and b) as well as c) and d) encode the same political claims: in b) the minister supports (+) the idea to give refugees vouchers instead of cash (claim category 212 2 ), and he proposes to extend the length of central accommodation of refugees (203). in a) both claims are opposed (-) by the left party (benefits in kind are regulated by the 'asylum seekers benefits act' in germany). analogously, in c) and d) the minister and his party cdu both express support to deport asylum seekers (claim 207). on a substantive level, the sentences are very similar. yet linguistically, they are quite different. while manifestos are written in direct speech and express the will of the party, the quotes from the public (newspaper) debate hold indirect speech that can be attributed to an individual actor. besides, they often differ in the lexical choices, e.g. colloquial vs. technical terms, as in "sent back" vs. "deport" in figure 1 c ) and d). to exacerbate matters, differences exist within genres in addition to the variability between them. the purpose of this paper is to determine whether computational models for claim detection in newspaper articles @xcite can be applied directly to party manifestos. our findings indicate that there is indeed a strong transferability and a substantive overlap between estimated party positions between the text types on the issue of migration. these findings are limited in scope to the german case and require further validation given the heterogeneity of both parties and policy issues. nevertheless, the contributions of our work address several points. first, we take a further step in the strive for scaling up the semi-automatic identification of tangible policy instruments in political texts @xcite by introducing cross2 we use the claim categories and codes for the migration debate proposed by @xcite . the data set can be found here: hdl.handle.net/11022/ 1007-0000-0007-db07-b and the coding scheme here: https://github.com/mardy-spp/mardy_ acl2019/blob/master/codebook.pdf text type evaluation. second, at a theoretical level, we provide further experimental insights into the relationship between party manifestos and newspaper debates: we analyse the substantive overlap of both formats considering the underlying claimdistribution per party in the two most recent federal elections in germany (2013 and 2017) and the interim public newspaper discourse @xcite to see whether previous findings from the political science literature (cf. section 2) hold. third, we extend the scope of our investigation to a concrete application scenario, namely a discourse network analysis of the debate at issue (right panel, figure 1 ). in this relational approach, the actor-claim-dyad is the first building block of a more complex, bipartite network structure. by abstracting from the actual text it is possible to directly contrast, compare, and even combine the two different formats on a network level. in this perspective, individual actors/parties and claims are two distinct types of vertices that are connected via edges that express support or opposition @xcite . we use this to demonstrate how our approach translates from newspaper data to party manifestos and how it leads to deeper conceptual insights. section 2 illustrates how our approach relates to the literature and compares characteristics of the distinct data sources. section 3 examines existing modeling approaches to claim identification. the results are presented in section 4 and 5 and their relevance is discussed in section 6. despite the mediating role played by newspapers in the diffusion process of disseminating party visions, vast differences between the text types exist on a conceptual level. while electoral programs paint a cohesive and unified picture of the party line, media coverage highlights conflict within parties. parties do not control the content written in newspaper articles. instead, their message is filtered and interpreted by editors and authors. in the news, parties compete with each other, trying to establish themselves and appeal to the reader @xcite . hence, existing studies measure and validate the substantive agreement between several (media) genres both in respect to party positioning on certain policy issues as well as the amount of attention (salience) these issues receive @xcite . for instance, comparisons are carried out between expert survey and party manifestos @xcite and, more recently, with the addition of newspaper articles as a third data source. @xcite find that while party positions are mostly congruent between manifestos and media coverage in the same election campaign, they differ when it comes to the salience of specific issues. thus, the media bias appears to have a stronger impact on the selection of certain topics than on the content's accuracy regarding party positions @xcite . one natural characteristic is that the substantive density (e.g. as a ratio of claims to text) is much higher in electoral programs than in newspaper articles. ## methodology in what follows, we spell out the two steps of our methodology underlying the experiments presented in the subsequent sections. ## experiments this section introduces the manifesto data (section 4.1) and presents results for fully automatic model performance in the cross-text type setting (section 4.2). we then carry out a more substantive comparison in terms of discourse networks and party positions that combines automatic analysis with manual post-processing (section 4.3). ## issue salience and party position in light of the promising performance, we now turn to the two remaining questions of substantive overlap between newspaper debate and party manifestos from the literature (cf. section 2). salience. the first one is: can we in fact observe a different focus in terms of salience between formats? figure 4 depicts the frequency distributions of (manually annotated) claim categories across public newspaper debate (red) and manifestos (blue), sorted by debate frequency. we observe considerable differences: popular categories during the 2015 debate are by no means also predominant in the party manifestos of 2013 and 2017. the correlation between genres amounts to only @xmath0.01 (@xmath1.91; for 2013 and 2017 alone @xmath2.28 and @xmath3.1, respectively), once one takes party specific salience into account. these results align well with findings from the literature on party salience. in contrast to the debatemanifesto split, the manually annotated distribution in the manifestos (blue) correlates extremely highly with the semi-automatic distribution (purple). (@xmath4.99, @xmath5.01), carrying over to individual parties as well, ranging from @xmath6.91 to @xmath7.98. this underlines again the usefulness of automatic analysis. positions. a final question is whether the positions of parties regarding certain categories are similar across formats and whether a semi-automatic approach captures complementary tendencies. positions (p ) can be calculated as the difference between all positive and negative referrals to a certain claim c divided by their sum @xcite : figure 5 shows the positioning of parties on different claims in the 2015 debates (x-axis) as well as the union of their 2013 and 2017 manifestos (yaxis), ranging from -1 (total opposition) to 1 (total support). this translates into a scale with the endpoints acceptance or rejection regarding a certain political demand with more moderate positions in between. the four uncoloured squares mark claims which were not automatically identified for this particular party in their manifesto. most observations are aligned perfectly across genres and located in two clusters (bottom left and top right corner). this indicates consistent party positions between text types (and also years) with occasional moderate changes and very few volte-faces. again, this confirms findings from the literature: the positions derived from manifestos are a strong indicator of positioning during discourse and vice versa (@xmath8.79, @xmath9.01; for 2013 and 2017 alone @xmath10.69 and @xmath11.83, respectively). unsurprisingly, we observe more variance in the position of parties during the debate (x-axis, figure 5 ), since it displays the aggregated and at times contradictory position of individual party members instead of an unified party. it is also noteworthy that 62% of party specific claims appear in one text type but not the other. this fact diminishes the party-specific overlap between formats quite a lot and also impacts the meaningfulness of the correlation. interestingly enough, the correlation between manually and semi-automatically estimated positions (excluding the missing claims) appears unaffected by this bias and amounts to @xmath12.99 (@xmath13.01). analogous to the application scenario in 4.3, this can to a great extend be explained by textual redundancy: assuming that parties emphasize claims multiple times and streamline their texts to avoid contradictions, we only need to find a single repetition of a specific claim to find the correct representation of that position. that makes a semi-automatic approach to positions much less demanding than to salience, because in the latter case every instance matters. in summary, while parties did not put the same emphasis on categories across text types, their standpoints appear to be consistent over the short time window under investigation. ## conclusion in this paper, we have explored whether computational models for claim detection trained on newspaper data could be applied to party manifestos as well. we found both high numeric performance, with f1 scores in the range of 0.80, and substantive overlap between manual and semi-automatically annotated sections in electoral programs. a first surprise was that the claim detection model achieved better results for the 2013 than for the 2017 manifesto. we have offered a machine learning-based explanation, and a set of three political ones: the first two were connected to deviant linguistic styles, and the third to the increasingly ubiquitous nature of the migration topic after the socalled 'refugee crisis' of 2015. these hypotheses can clearly benefit from a future more systematic re-examination on a larger scale. assessing the perspectives for building discourse networks from the party manifestos, we were able to semi-automatically identify 93% of the edges by taking advantage of textual redundancy. with this in mind, we were able to confirm existing findings from the political science literature on party position and issue salience. while the positions of different parties on distinct claims are mostly congruent between formats, salience turned out to be largely uncorrelated. clearly, our current approach has further limitations. first, we only focus on one of many potential policy issues (migration) within a very short time-frame. this begs the question of generalizability, which we aim to address in three ways in future research: concretely, ongoing work extends our analysis to different years @xcite , to the issue of pensions, and to another newspaper to validate our findings as well as to account for issue-specific characteristics (such as the degree of polarization or the granularity of proposed policy instruments). second, even though we already reduced it considerably, we still rely on human intervention for the semi-automatic annotation. yet, from a mixedmethods perspective, this approach highlights the mutual benefit of interleaving political science and nlp. on the one hand the cross-validation of discourse networks with networks based on electoral programs could open up new venues for dna and, conversely, explore the role public discourse plays in the formation of these programs. on the other hand, a still open question is whether training and test data can be reversed. to answer this question additional labeled material is needed. positive results might prove as a welcome shortcut to the creation of comprehensive codebooks for political claims analysis given the much higher ratio of claims to text in manifestos. both approaches would require computational assistance and thus enable further integration of nlp and css.