Improving Neural Political Statement Classification with Class Hierarchical Information (2022)

Abstract: Many tasks in text-based computational social science (CSS) involve the classification of political statements into categories based on a domain-specific codebook. In order to be useful for CSS analysis, these categories must be fine-grained. The typically skewed distribution of fine-grained categories, however, results in a challenging classification problem on the NLP side. This paper proposes to make use of the hierarchical relations among categories typically present in such codebooks:e.g., markets and taxation are both subcategories of economy, while borders is a subcategory of security. We use these ontological relations as prior knowledge to establish additional constraints on the learned model, thusimproving performance overall and in particular for infrequent categories. We evaluate several lightweight variants of this intuition by extending state-of-the-art transformer-based textclassifiers on two datasets and multiple languages. We find the most consistent improvement for an approach based on regularization.

## introduction the argumentative or discursive turn in policy analysis and political science more generally has long established the value of textual sources for the analysis of politics and policies @xcite . traditionally, data sources such as interviews or newspaper reports were annotated using various methods of qualitative text analysis @xcite . at the heart of this analysis is always a codebook, i.e., guidelines that map actual statements or textual passages to the abstract concepts relevant for the respective research. categories in codebooks are almost always arranged hierarchically, with fine-grained categories being grouped together into supercategories that are often, but not always, more abstract. finegrained categories are generally generated inductively from the analyzed texts in an iterative pro-cess of summarizing and abstracting from the original text, while the supercategories are deductively generated from existing knowledge of the relevant policy field and from theoretical and conceptual findings of prior research. for example, the codebook of the long-running comparative manifesto project (cmp), which analyzes party manifestos across several countries, includes 7 supercategories (such as external relations or economy) with 56 subcategories: for economy, among others, free market, market regulation, economic goals, etc. @xcite . here, supercategories represent the separation of policy fields that is reflected in political institutions, e.g., ministries. fine-grained, hierarchical schemes help researchers both with data annotation and with analysis. annotation is often easier when the annotation decision is (implicitly) first based on a supercategory and then on fine-grained subcategories. for analysis, supercategories structure the annotated material according to different levels of abstraction, thereby supporting interpretation and modeling. while such a hierarchical process a natural choice in manual annotation, the situation is different when we move to (semi)-automatic analysis in nlp: due to the large number of fine-grained subcategories, the available data is distributed among many categories. in addition, most categories are infrequently attested, since categories typically show a skewed distribution. this makes for a difficult classification problem, and existing prediction studies have often only addressed the more coarsegrained supercategory level @xcite @xcite . in this study, we ask whether we can use the hierarchical structure of political science codebooks to our advantage: knowing that two subcategories (as free market and market regulation) belong to the same supercategory (economy) could lead us to expect that the representations learned for these categories should be more similar to one another than to categories that belong to other supercategories. in this manner, the representations learned for smaller categories can be biased in the right direction by their larger neighbor categories. this paper makes the following contributions: this paper builds on an earlier study of ours @xcite , whose scope is extended in multiple dimensions. at the phenomenon level, we broaden the focus from (forward-looking) political claims to (general) political statements. at the methodological level, we propose an ontology of methods for encoding hierarchical information. at the experimental level, we now take into consideration two text types involving five different languages. the code, models and dataset splits used in this study are available at https: //www.ims.uni-stuttgart.de/data/inpsc . codebooks for political statement categorization codebooks used in large-scale annotation projects cover a broad variety of research interests and text types. yet, regardless of whether they have been created to analyze political party manifestos @xcite , political statements in the european public sphere @xcite , legitimation discourses about political and economic regimes @xcite , or the migration debate in germany @xcite , they all group their categories of interest into a limited number of supercategories which reflect the existing research in the respective field. text classification automatic political statement classification is fundamentally a text classification task on relatively short texts, with the class inventory given by the codebook. depending on the properties of the annotation, the task is either single-label or multi-label text classification. in single label text classification, each text is assigned exactly one label, which is used in nlp applications where the labels are mutually exclusive, such as in entailment or stance detection @xcite @xcite . in contrast, multi-label text classification assigns any number of categories to a text, which is better suited for tasks where the categories are overlapping or describe complementary aspects, e.g. topic categorization @xcite @xcite . currently, transformerbased models @xcite represent the current state of the art for text classification in general @xcite and political statement classification in particular @xcite . a number of studies have investigated ways to integrate hierarchical information into classification. a first family of approaches develops dedicated architectures such as capsule networks @xcite or encoders of the hierarchies @xcite . these models are typically trained end-to-end, which requires amounts of data that are rarely available in css. we focus on lightweight approaches compatible with fine-tuning, described in section 3. ## conclusion this paper addresses the task of political statement classification focussing on the challenge of class imbalance. we have argued that the hierarchically structured codebooks developed by political science projects are a source of domain knowledge that can be integrated in classification models. we extend state-of-the-art transformer models with lightweight modules that implement this intuition in different ways. we evaluate on two datasets, covering two codebooks, single-label and multilabel classification, and various languages. our main findings are robust across the different setups: inclusion of hierarchical information virtually always improves classification, and the methods we consider are sufficiently complementary that their benefits combine. we obtain improvements even for fairly large datasets, with diminishing benefits for very large datasets -which is plausible, given that performance improves particularly for low-frequency categories. the latter finding -strong improvements for low-frequency categories -is arguably important with regard to algorithmic fairness @xcite , since in the case of rare categories, a small number of prediction errors is sufficient to substantially impact the reliability of downstream analyses. indeed, multi-ple causes of low frequency categories exist. as one example, in analyses over time, statement frequencies co-vary naturally with topic prominence, and analyses like the (semi-)automatic extraction of network representations to assess dynamics of political debates @xcite ) may misrepresent the contribution of infrequent categories. as another example, work on the framing of immigration discourse on twitter @xcite has shown that employing issue-specific categories (e.g., "victim:war", "victim: discrimination", "threat:jobs", "threat:public order") reveal ideological and regional patterns which would be missed by the commonly employed generic frames such "economy" or "morality" @xcite -but at the cost of introducing many fine-grained categories which are sparse and attested with widely different frequencies. our work demonstrates that a well designed hierarchical codebook, combined with the right computational devices, can go a long way towards redressing the challenges that arise from this situation. an more detailed assessment of the impact of our methods on downstream tasks remains future work.