Strategies for political-statement segmentation and labelling in unstructured text (2025)

Abstract: Analysis of parliamentary speeches and political-party manifestos has become an integral area of computational study of political texts. While speeches have been overwhelmingly analysed using unsupervised methods, a large corpus of manifestos with by-statement political-stance labels has been created by the participants of the MARPOR project. It has been recently shown that these labels can be predicted by a neural model; however, the current approach relies on provided statement boundaries, limiting out-of-domain applicability. In this work, we propose and test a range of unified split-and-label frameworks—based on linear-chain CRFs, fine-tuned text-to-text models, and the combination of in-context learning with constrained decoding—that can be used to jointly segment and classify statements from raw textual data. We show that our approaches achieve competitive accuracy when applied to raw text of political manifestos, and then demonstrate the research potential of our method by applying it to the records of the UK House of Commons and tracing the political trajectories of four major parties in the last three decades.

## introduction among the genres used by politicians to communicate with each other and voters, two of the most important ones are party manifestos and speeches made in deliberative assemblies, such as the house of commons in the uk or the bundestag in germany. these sources are publicly available, but their sheer volume makes manual analysis of them a very challenging task. as such, the study of party manifestos and parliamentary debates has become one of the cornerstones of computational analysis of political texts (cf., among others, fišer et al., 2022; arroyo, 2022; múller and proksch, 2023). @xcite the authors contributed equally to this paper; the order is alphabetical. while members of the research community share an interest in analysing the stances expressed by politicians towards different issues, the particular approaches taken for these two types of texts have largely differed. the analysis of party manifestos has, to a large extent, coalesced around the labelling scheme developed in the framework of the marpor project (volkens et al., 2021) and used to manually annotate manifestos from more than 60 countries, written in almost 40 languages. foot_0 marpor labels are attached to statements, semantically coherent units on the sentence or sub-sentence level. these labels correspond to political issues, such as national defence or migration, but often encompass both an issue and a particular stance towards that issue. for example, label 504, 'welfare state expansion', is assigned to 'favourable mentions of need to introduce, maintain or expand any public social service or social security scheme.' therefore, by means of simply counting different labels assigned to statements from a particular manifesto, it is possible to obtain a rather fine-grained representation of the political program expressed therein. until recently, efforts to assign these labels automatically had been largely unsuccessful and limited in scale (dayanik et al., 2022). it was subsequently shown by nikolaev et al. (2023) foot_1 that contemporary multilingual models can be used for adequate cross-lingual analyses. however, their approach relies on the availability of statement boundaries, not provided by existing nlp tools, which limits the practical applicability of the trained models. @xcite this situation stands in contrast to the study of parliamentary debates, where labelled corpora are non-existent and the basic unit is usually a whole speech. analysis in this domain has overwhelmingly relied on unsupervised exploratory methods, such as topic modelling, or even manual analysis, and targeted simple binary categories and aggregate scales (abercrombie and batista-navarro, 2020; nanni et al., 2022; skubic and fišer, 2024). @xcite the marpor categorization scheme has proven to be a powerful tool for political-text analysis, applicable to almost any text in this domain, @xcite and the fact that labelled data and models trained on them only exist for party manifestos is largely a technical obstacle. therefore, in this work we aim to solve the problem of projecting the marpor annotations to any running text. in order to do this, we experiment with a series of models, spanning the landscape of transformer-based architectures. as a first step, we replace the encoder-based statement-level classifiers proposed by nikolaev et al. (2023) and burst et al. (2023b) with a linear-chain crf layer (lafferty et al., 2001) that learns to predict statement boundaries jointly with marpor labels using raw manifesto texts. this pipeline is very memory efficient and provides quick training and inference. however, its ability to understand label sequences is limited by the expressive power of linear-chain crfs, motivating investigation of autoregressive models. as a more expressive but also more computationally demanding alternative, we propose using a pre-trained t5-family model that is fine-tuned to split large textual chunks into statements and label these statements at the same time. finally, we try to solve the task by using incontext learning, i.e. forgoing fine tuning and providing labelled examples during inference with a state-of-the-art decoder-only model. @xcite we show that, even though fine-tuned t5-type models produce the best in-domain results, their @xcite a limited attempt at applying the marpor coding scheme to parliamentary data, again on the speech level, has been made by abercrombie and batista-navarro (2022), but it relies on a rather strong assumption that the whole speech revolves around the same narrow topic. @xcite cf. an analysis of judges' decisions using this framework by rosenthal and talmor (2022). @xcite showed that using long-input berttype model for directly predicting a scaling score, rile, produced bad results, and it seems that language models is in general poorly suited for regression. therefore in this study we only experiment with statement-level classification, which has additional practical benefits. high computational demands and slow inference limit their practical applicability to large-scale outof-domain experiments. for such cases, the crfbased model makes for a better choice, showing a slight performance degradation on in-domain evaluation but orders-of-magnitude faster inference. equipped with our crf model, capable of efficiently segmenting and labelling statements from raw text, we perform an exploratory analysis of the uk parliamentary records. @xcite we further discuss the problem of the parliamentary data being out-ofdomain, especially in terms of label sequences, and propose to mitigate it using model ensembling. ## data we target the same original-language and translated subsets of the marpor dataset as used by nikolaev et al. (2023). @xcite . this dataset, a subset of the full collection marpor-labelled manifestos, comprises a total of 1314 manifestos from 41 different countries, with the untranslated texts representing 27 different languages. manifestos are segmented into claims and labelled, with each manifesto averaging just over 1000 claims, and labels ranging across the 143 marpor claim categories. nikolaev et al. (2023) explored two evaluation settings for this task: leave-one-countryout, a cross-validation strategy where each crossvalidation split held out a single country as a test partition, and old-vs.-new, wherein pre-2019 manifestos were used as a training set and post-2019 manifestos were used for evaluation. in this work, we adopt the leave-one-country-out setting, as it is more challenging. since training and testing larger models on all 41 countries from their dataset is not practicable, we adopt the following approach: after a complete preliminary analysis done using the xlm-r + crf approach, we split the countries into quartiles based on the test-set performance. we then select a country from the middle of the each quartile and used this country's manifestos as a test set for subsequent experiments. for each of the test countries we also use the same set of dev-set-countries' manifestos when training the crf and fine-tuning flan t5. @xcite the dataset for the exploratory analysis of parliamentary records is described in § 6. ## methods the problem of jointly segmenting and classifying statements from text is an example of a span identification, or extraction, task. in spite of the fact that all models we use rely on the same underlying transformer architecture, they demand different approaches to task operationalisation and input/output encoding. we specify them below. @xcite 3.1 crf input formatting. following standard practice, we encode the statements using the bio scheme (ramshaw and marcus, 1995) and use a sequencelabelling model to predict token-wise labels. as the spans we are extracting form a total cover of our texts, the o label is ultimately only used for padding and bos/eos tokens. the architecture. our model combines a linearchain crf with a pre-trained xlm-roberta (xlm-r) encoder (conneau et al., 2019) providing token-wise emission scores for the crf. due to xlm-r's multi-lingual pre-training, we are able to directly use manifestos in their original languages. as the political manifestos we train on are significantly larger than our encoder's context window, we divide the input text into multiple overlapping windows, feed these windows to our encoder independently, and stitch together the contextualized representations obtained from the centre of each window for use as input to the crf. in this way, we can process sequences of arbitrary length, while still ensuring that each token's representation was generated with adequate context to both left and right. during inference, we feed entire documents as input to our model in this manner, irrespective of length, while during training, for performance reasons, we limit model inputs to 1024 tokens, yielding a maximum of four overlapping windows. a complete description of our model, including hyperparameters and splitting procedures, is provided in appendix b. training. for each cross-validation split, we initialize our encoder with pre-trained xlm-r tria, czech republic). @xcite the training code for the study will be uploaded to a public repository in case of acceptance. ## results with the leave-one-country-out cross-validation setting, we obtain one set of model predictions for each country. for the crf-based model, where all 41 countries were processed, we analyse our results on the union of by-country predictions. for flan t5 xl and llama, we report the results for each test country individually. ## discussion of computational demands in this section, we contrast computational demands of different approaches. we show that while training demands of even bigger models that we use are manageable, given access to typical research-grade infrastructure, inference on them becomes limited to hundreds, at most thousands of examples, which limits their applicability to larger corpora in computational political science numbering millions of data points. ## analysis of parliamentary debates we now turn to the analysis of parliamentary data to show how our raw-text-capable crf-based model can be applied in another domain. while it is likely less powerful than fine-tuned flan t5 xl, it is incomparably faster in inference and can be used to process large corpora without access to massive computational resources. ## related work as far as we are aware, no prior work addresses the problem of assigning marpor labels to raw text, and the efforts were focused on providing higher-level stance or scaling analyses. for example, subramanian et al. (2018) provided manifestolevel scaling scores by aggregating over lstmbased representations of sentences and taking into account historical rile values, while liu et al. (2022) present a model for determining ideology and stance, where both target values are encoded as binary or 3-element scales. the problem of automatically assigning contentful marpor labels to statements in party manifestos was first addressed on a smaller scale by dayanik et al. (2022) and ceron et al. (2023), and then in a larger cross-lingual setting by nikolaev et al. (2023) and burst et al. (2023b,a). all these studies assumed, however, that gold statement boundaries are provided, which contrasts with the fact that many marpor statements consist of sub-sentences, which demands a dedicated spanextraction module. the necessity of completely splitting the input into sub-sentence-level chunks contrasts our setting with span-extraction tasks, such as ner, and more straightforward sentence-segmentation settings, where the need for domain-specific approaches has also been recognised. in the latter area, crf-and encoder-based approaches continue to demonstrate strong results, cf. ## conclusion the analysis of political texts has long been impeded by the absence of a model providing identification and fine-grained semantic labelling of statements. in this work, we show that it is possible to assign statement boundaries and stance labels at the same time. using well-proven methods, a bert-type encoder with a crf layer, we reach good performance on the manifesto data and then 8 demonstrate that our model can provide insightful analyses of parliamentary data in the standard marpor framework. we furthermore show that better results can potentially be attained using simple fine-tuning of a large text-to-text model, but its low inference speed precludes its use for largescale exploratory studies. finding ways of accelerating inference on high-volume raw-text segmentation and analysis is an important avenue for future work.