Towards Quantifying Magnitude of Political Bias in News Articles Using a Novel Annotation Schema (2021)

Abstract: Media bias is a predominant phenomenon present in most forms of print and electronic media such as news articles, blogs, tweets, etc. Since media plays a pivotal role in shaping public opinion towards political happenings, both political parties and media houses often use such sources as outlets to propagate their own prejudices to the public. There has been some research on detecting political bias in news articles. However, none of it attempts to analyse the nature of bias or quantify the magnitude ofthe bias in a given text. This paper presents a political bias annotated corpus viz. PoBiCo-21, which is annotated using a schema specifically designed with 10 labels to capture various techniques used to create political bias in news. We create a ranking of these techniques based on their contribution to bias. After validating the ranking, we propose methods to use it to quantify the magnitude of bias in political news articles.

## introduction an increase in internet and social media access has made it easier for people to know the happenings from all parts of the world. this improved reach of the media has led to people and organisations utilising such platforms to increase their popularity. with this extensive usage of mass communication, there is an increased risk of misinformation, fake news, twisting of facts, and the spread of controversies. in such cases, it is the responsibility of the media houses to verify the authenticity of the reported information and ensure that it is complete, neutral, and not misleading. however, we often see instances where some news articles emphasize more on particular viewpoints selectively. in some cases, journalists and media houses either intentionally or subconsciously present biased information aligned with their own political ideologies. this can alter public opinion hugely and impact the decision of the general public. this phenomenon of political bias is very prevalent in democracies, especially during elections, where influencing public opinion can have far-reaching consequences. detection of political bias in the news is a complex task when it comes to a multi-party political system like india, where multiple parties operate at both state and national levels 1 . there has been limited work to date in this area, as discussed in section 2. however, in a scenario where most of the political content contains bias in some form, it is not just sufficient to identify the existence of bias. therefore, we propose two important contributions through this paper -a fine-grained annotation schema specifically designed for analysing political bias and a new sentiment analysis based approach to quantify the magnitude of political bias in an article. we also present pobico-21, an annotated corpus for political bias, containing 500 news articles along with their headlines annotated for various aspects of political bias using our novel annotation schema as discussed in section 3. the dataset contains news articles in telugu foot_1 , which is a low resource dravidian language spoken primarily in the states of telangana and andhra pradesh in india. ## related work the area of political discourse analysis is closely associated with media studies, political science, anthropology, sentiment analysis, and opinion mining. however, most of the work done is limited to english, primarily due to the high availability of data. for example, @xcite identify subjective sentences in blog posts to understand the political orientation of a given text. @xcite proposed a rule-based linguistic method to identify bias in a news article based on the semantics of the headline and the article. 2020 ) used attention mechanism on different kinds of document representations to identify bias in an article. @xcite proposed a hybrid approach to improve the performance of such ml models by establishing a correlation between presupposition and bias. @xcite pointed out that restricting to lexical level for detecting bias might not be sufficient and put forward a method of identifying spans of bias by looking at sentences/clauses which are speculative or tangential to the main point being talked about. @xcite made use of the comments and opinions of readers in order to classify articles as liberal or conservative. but the limitation to this approach is that it cannot perform well in a political setting where multiple alliances and several parties operate at different levels of the government. ## data and annotation our first requirement was a dataset of political news articles containing positive and negative bias along with some unbiased articles. for this purpose, we used the public dataset created by @xcite as our primary data to begin with. our aim is to annotate the dataset with different techniques through which bias is generated, and identify the levels of reporting at which it happens. this will be discussed in detail in section 3.1. we removed the articles which were very short, or had no clear target of bias, either positive or negative. the statistics of our dataset are presented in the reason for relatively fewer unbiased articles in the corpus is due to the ubiquitous existence of some level of bias in most political articles. this can happen because of the affiliation of media houses with political parties -such as the sakshi 3 3 https://en.wikipedia.org/wiki/sakshi_ telugu newspaper owned by the family of the y.s. jagan, the leader of ysrcp party and current chief minister of andhra pradesh. other possible reasons include the incorporation of subjectivity, or the influence of the personal ideologies of the journalists, editors, and writers of an article, despite the ideal expectation of objective reporting. ## schema validation to test the reliability of the schema proposed in section 4.2, we tested it on a bias magnitude comparison task against the human annotation results. for this purpose, we created a test dataset consisting of 100 article pairs. for each pair, the task was to identify the more biased article of the two. all the articles were taken from the political news domain. for human annotation, we had the annotation done by two annotators who had no idea about the schema. we removed the articles on which there was a disagreement between the two annotators. we were left with 92 articles. then we gave the test pairs to two other annotators who had the list of the 10 techniques along the definition, explanation and examples. their task was to identify all the techniques present in the two articles given in each article pair. then they were instructed to give bias scores for each article by adding the scores of all the techniques from if both the articles in a pair had same bias score but at least one non-common label, the comparison was to be done based on the non-common label. if two articles had the same score and same labels, we asked the annotators to mark the shorter of the two articles as the more biased one. we found that 66 out of 92 times, we were able to correctly identify the more biased article using our ranking. this gives us confidence that the ordering of techniques done through our ranking has a strong correlation with the relative contribution of each technique towards political bias in an article. ## conclusion and future work this paper explores the creation of a novel annotation schema which captures the nuances of political bias in a fine grained manner. we propose a sentiment analysis based ranking of 10 political bias techniques, and a validation study to show that this ranking corresponds to the relative contribution of each of these techniques towards political bias. we also contribute a fully annotated dataset-pobico-21, containing 500 articles annotated using this schema 10 . the fine grained annotation can also be used for making other interesting observations from political news data such as identifying media houses which often use biased/misleading headlines, mining frequently co-occuring techniques of political bias, etc. figure 4: process workflow of the proposed system of bias scoring 6.1 proposed direction for future research in the future, we can create a end-to-end bias scoring system which takes an article as an input, tokenises it, detects all potential spans of text conatining political bias, assigns the relevant political bias techniques to each span, and computes the final score as the summation of each span length multiplied by the relative sentiment scores of the techniques present in that span. this is shown in a step-by-step manner with an example in figure 4 . detection of bias spans can be treated as a standard sequence labelling problem. with good amount of labelled data, we can use neural transformer architectures like bert @xcite , or its variants like roberta @xcite , @xcite etc. once that is done, we can do a one-to-many mapping from