Fine-grained Classification of Political Bias in G erman News: A Data Set and Initial Experiments (2021)

Abstract: We present a data set consisting of German news articles labeled for political bias on a five-point scale in a semi-supervised way. While earlier work on hyperpartisan news detection uses binary classification (i.e., hyperpartisan or not) and English data, we argue for a more fine-grained classification, covering the full political spectrum (i.e., far-left, left, centre, right, far-right) and for extending research to German data. Understanding political bias helps in accurately detecting hate speech and online abuse. We experiment with different classification methods for political bias detection. Their comparatively low performance (a macro-F1 of 43 for our best setup, compared to a macro-F1 of 79 for the binary classification task) underlines the need for more (balanced) data annotated in a fine-grained way.

## introduction the social web and social media networks have received an ever-increasing amount of attention since their emergence 15-20 years ago. their popularity among billions of users has had a significant effect on the way people consume information in general, and news in particular @xcite . this development is accompanied by a number of challenges, which resulted in various nlp tasks that deal with information quality @xcite @xcite . due to the data-driven nature of these tasks, they are often evaluated under the umbrella of (un)shared tasks, on topics such as rumour detection or verification @xcite , offensive language and hate speech detection @xcite @xcite @xcite @xcite or fake news and fact-checking @xcite @xcite . several shared tasks concentrate on stance @xcite and hyper-partisan news detection @xcite , which predict either the stance of the author towards the topic of a news piece, or whether or not they exhibit allegiance to a particular party or cause. we argue that transparency and de-centralisation (i. e., moving away from a single, objective "truth" and a single institution, organisation or algorithm that decides on this) are essential in the analysis and dissemination of online information @xcite . the prediction of political bias was recently examined by the 2019 hyperpartisan news detection task @xcite with 42 teams submitting valid runs, resulting in over 30 publications. this task's test/evaluation data comprised english news articles and used labels obtained by @xcite , but their five-point scale was binarised so the challenge was to label articles as being either hyperpartisan or not hyperpartisan. we follow @xcite in claiming that, in order to better understand online abuse and hate speech, biases in data sets and trained classifiers should be made transparent, as what can be considered hateful or abusive depends on many factors (relating to both sender and recipient), including race @xcite , gender @xcite , and political orientation @xcite . this paper contributes to the detection of online abuse by attempting to uncover political bias in content. we describe the creation of a new data set of german news articles labeled for political bias. for annotation, we adopt the semi-supervised strategy of @xcite who label (english) articles according to their publisher. in addition to opening up this line of research to a new language, we use a more fine-grained set of labels. we argue that, in addition to knowing whether content is hyperpartisan, the direction of bias (i. e., left-wing or rightwing) is important for end user transparency and overall credibility assessment. as our labels are not just about hyperpartisanism as a binary feature, we refer to this task as political bias classification. we apply and evaluate various classification models to the data set. we also provide suggestions for improving performance on this challenging task. the rest of this paper is structured as follows. section 2 discusses related work on bias and hyperpartisanism. section 3 describes the data set and provides basic statistics. section 4 explains the methods we apply to the 2019 hyperpartisan news detection task data (for evaluation and benchmarking purposes) and to our own data set. sections 5 and 6 evaluate and discuss the results. section 7 sums up our main findings. ## data collection and processing we obtain our german data through two different crawling processes, described in sections 3.1 and 3.2, which also explain how we assign labels that reflect the political bias of the crawled, german news articles. since the 2019 shared task data which we use for benchmarking purposes is downloaded and used as is, we refer to @xcite for more information on this data set. ## methodology in this section we describe the different (feature) representations of the data we use to train different classification models on as well as our attempts to alleviate the class imbalance problem ( ## discussion in many nlp tasks, the strategy of using bert as a language model that is fine-tuned to a specific the lists of words show that the majority of words for the far-left classification are neither exclusively nor mainly used by left-wing news media in general, e. g., wirkt (works), seither (since) or geliebte (beloved, lover). an exception is antisemitische (anti-semitic), with anti-semitism in society being a common topic in left-wing media. other highlighted words are likely to be related to the topic of refugee migration and its causes, such as hungernden (hungry people) and sahelzone (sahel), an area known for its conflicts and current societal challenges. in contrast to the words we identified for the far-left, we found most of the words we identified for the far-right to be more descriptive of this side of the political spectrum. nearly all words listed under far-right in generally, entries in ## conclusion and future work we present a collection of german news articles labeled for political bias in a semi-supervised way, by exploiting the results of a survey on the political affiliation of a list of prominent german news outlets. 16 this data set extends on earlier work on political bias classification by including a more fine-grained set of labels, and by allowing for research on political bias in german articles. we propose various classification setups that we evaluate on existing data for benchmarking purposes, and then apply to our own data set. our results show that political bias classification is very challenging, especially when assuming a non-binary set of labels. when using a more fine-grained label set, we demonstrate that performance drops by 36 points in accuracy, from 79 in the binary case to 43 in the more fine-grained setup. political orientation plays a role in the detection of hate speech and online abuse (along with other dimensions, such as gender and race). by making available more data sets, in different languages, and using as many different publishers as possible (our results validate earlier findings that models quickly over-fit to particular publisher identity features), we contribute to uncovering and making transparent political bias of online content, which in turn contributes to the cause of detecting hate speech and abusive language @xcite . while labeling articles by publisher has the obvious advantage of producing a larger number of labeled instances more quickly, critical investigation and large-scale labeling of individual articles must be an important direction of future work.