Using Hierarchical Class Structure to Improve Fine-Grained Claim Classification (2021)

Abstract: The analysis of public debates crucially requires the classification of political demands according to hierarchical claim ontologies (e.g. for immigration, a supercategory “Controlling Migration” might have subcategories “Asylum limit” or “Border installations”). A major challenge for automatic claim classification is the large number and low frequency of such subclasses. We address it by jointly predicting pairs of matching super- and subcategories. We operationalize this idea by (a) encoding soft constraints in the claim classifier and (b) imposing hard constraints via Integer Linear Programming. Our experiments with different claim classifiers on a German immigration newspaper corpus show consistent performance increases for joint prediction, in particular for infrequent categories and discuss the complementarity of the two approaches.

## introduction newspaper articles are an invaluable source for the analysis of public debates. in political science, it is common to manually annotate the articles by identifying claims (text spans which report a demand on a specific policy aspect), assigning them fine-grained claim categories from domain-specific claim ontologies and attributing them to actors (e.g., politicians or parties). actors and claim categories together can be used to construct expressive discourse networks @xcite for in-depth analysis of debate structure and dynamics. in line with the trend of applying nlp methods to questions from political science (e.g., @xcite claim classification has been framed as generic text classification @xcite . that study however addresses only coarse-grained categories and reports mixed results even at that level, with a macro f1 of 46. this is arguably due to the well-known problems of fine-grained classification: the larger the set of classes, the more data would be desirable, while in actuality, the number of instances per class shrinks @xcite . our paper aims at developing practically useful models of fine-grained claim classification. its main proposal is to exploit the hierarchical nature of claim ontologies by jointly predicting (frequent) supercategories and (informative) subcategories. by enforcing consistency between the levels, the predictions can profit off each other. we experiment with two operationalizations of this idea. the first one, hierarchical label encoding @xcite ) introduces "soft" constraints through parameter sharing between classes in the classifier. the second one, integer linear programming (ilp, e.g., @xcite ) introduces "hard" constraints in a post-processing step. both methods can be applied to a range of claim classifier architectures. we present experiments with four architectures on a german manually annotated corpus from the so-called refugee crisis in germany in 2015. we answer the following questions: do hle and ilp improve the performance in our experimental setup? (yes.) is there complementarity between them? (yes.) does the effect depend on the underlying architectures. cal science experts for the migration domain. the corpus contains 3827 annotated textual spans, each of which is assigned one or more categories from the claim ontology described below: spans can be assigned multiple categories when the statements touch on more than one policy issue. claim ontology policy debates are inherently complex, as a reflection of the complexity of the problems which the policy addresses: in our case, control of migration, but also integration of refugees, foreign policy, etc. in our case, the claim ontology consists of 100 subcategories which are grouped into 8 supercategories (cf. ## basic claim classification given the properties described above, we model claim classification as multi-label classification. we follow previous work on coarse-grained claim classification @xcite in comparing a set of neural models, ranging from baselines to stateof-the-art architectures. all models are trained using cross entropy loss with the sigmoid activation function. all models except bert use custom fast-text (bojanowski et al., 2017) word embeddings pretrained on a german newswire corpus. foot_1 lstm this model passes the input through a single-layer lstm. the final hidden state is used as input to a fully connected layer. bilstm a single-layer bidirectional lstm @xcite traverses the input. the final hidden states in both directions are concatenated and fed to a fully connected layer. bilstm+attention this model combines the bilstm architecture with the attention mechanism described in @xcite . the input is fed to a single-layer bilstm. then, the attention-weighted sum of the hidden states corresponding to the input sequence is fed to a fully connected layer. bert this is a pretrained bert @xcite model trained solely on german corpora foot_2 and a fully connected layer which is trained while the bert encoder is fine-tuned. after each input is encoded, we use the final hidden state of the first token, corresponding to the special token @xcite , as the contextualized representation of the input which serves as input to a fully connected layer. ## integrating hierarchical class structure the obvious shortcoming of the model architectures sketched above is that they make the standard assumption of class independence -even though we know that the classes in claim classification are related. we therefore build on the idea that we can label all documents with both sub-and supercategories during training time, and then encourage the model to jointly predict categories at both levels so that these predictions are consistent with one another. the expectation is that this creates an incentive to learn better representations for the fine-grained classes. we now sketch two generally applicable methods that implement this idea. ## experimental evaluation setup. we remove very infrequent subcategories in the dataset by applying a threshold of 20 instances. smaller categories are merged with the preexisting subcategory x99, which exists for each supercategory as a 'catch-all' category for outlier cases. after filtering, there are 8 super-and 72 subcategories left in the dataset (cf. main results. the addition of ilp (2nd column) leads to inconsistent changes in precision but always yields better recall and f-scores. lstm and bilstm still perform significantly worse than the other two models. when we switch to hle, all metrics for all models are boosted significantly, showing that parameter sharing via the super/sub-category co-occurrence matrix is a successful across the board. we observe the largest improvement for bert, where hle yields an improvement of 12 points in f1, and leads to the overall highest precision (0.75). the last column (hle + ilp) shows a substantial complementarity of the two methods: models consistently improve over both the hle only and ilp only setting. specifically, hle+ilp models achieves better recall scores than hle models (+7 points on average) and better precision (+8 points on average) scores than ilp models. the effect is least pronounced for the best architecture (bert); nevertheless, bert with hle and ilp achieves the overall highest recall (0.59) and f-score (0.60), corresponding to an improvement of 13 points f1 compared to the 'plain' version. the fact that the f1 boost is fueled mainly by recall is particularly promising because optimizing for recall is the best strategy when nlp tools are employed in semiautomatic annotation @xcite . ## conclusion this paper has identified automatic fine-grained claim classification as a crucial, but underaddressed, component of political discourse analysis. we have demonstrated that hierarchical class structure can be exploited to lift fine-grained claim classification to a usable level, showing robust improvements even for transformer architectures and in particular for low-frequency claim categories. addressing the low-frequency issue is particularly relevant in the broader context of the goals of political science. political discourse unfolds over time, and every prominent issue starts out as infrequent. the true dynamics of debates can only be captured if the classifiers are able to pick up the less salient categories @xcite . future work involves investigating these concerns on a wider range of datasets, as well as evaluating fine-grained claim classification for semi-automatic discourse network construction.