We Can Detect Your Bias: Predicting the Political Ideology of News Articles (2020)

Abstract: We explore the task of predicting the leading political ideology or bias of news articles. First, we collect and release a large dataset of 34,737 articles that were manually annotated for political ideology –left, center, or right–, which is well-balanced across both topics and media. We further use a challenging experimental setup where the test examples come from media that were not seen during training, which prevents the model from learning to detect the source of the target news article instead of predicting its political ideology. From a modeling perspective, we propose an adversarial media adaptation, as well as a specially adapted triplet loss. We further add background information about the source, and we show that it is quite helpful for improving article-level prediction. Our experimental results show very sizable improvements over using state-of-the-art pre-trained Transformers in this challenging setup.

## introduction in any piece of news, there is a chance that the viewpoint of its authors and of the media organization they work for, would be reflected in the way the story is being told. the emergence of the web and of social media has lead to the proliferation of information sources, whose leading political ideology or bias may not be explicit. yet, systematic exposure to such bias may foster intolerance as well as ideological segregation, and ultimately it could affect voting behavior, depending on the degree and the direction of the media bias, and on the voters' reliance on such media @xcite @xcite . thus, making the general public aware, e.g., by tracking and exposing bias in the news is important for a healthy public debate given the important role media play in a democratic society. media bias can come in many different forms, e.g., by omission, by over-reporting on a topic, by cherry-picking the facts, or by using propaganda techniques such as appealing to emotions, prejudices, fears, etc. @xcite bias can occur with respect to a specific topic, e.g., covid-19, immigration, climate change, gun control, etc. @xcite it could also be more systematic, as part of a political ideology, which in the western political system is typically defined as left vs. center vs. right political leaning. predicting the bias of individual news articles can be useful in a number of scenarios. for news media, it could be an important element of internal quality assurance as well as of internal or external monitoring for regulatory compliance. for news aggregator applications, such as google news, it could enable balanced search, similarly to what is found on allsides. 1 for journalists, it could enable news exploration from a left/center/right angle. it could also be an important building block in a system that detects bias at the level of entire news media @xcite @xcite , such as the need to offer explainability, i.e., if a website is classified as left-leaning, the system should be able to pinpoint specific articles that support this decision. in this paper, we focus on predicting the bias of news articles as left-, center-, or right-leaning. previous work has focused on doing so at the level of news media @xcite or social media users @xcite , but rarely at the article level @xcite . the scarce article-level research has typically used distant supervision, assuming that all articles from a given medium should share its overall bias, which is not always the case. here, we revisit this assumption. our contributions can be summarized as follows: the rest of this paper is organized as follows: we discuss related work in section 2. then, we introduce our dataset in section 3, we describe our models for predicting the political ideology of a news article in section 4, and we present our experiments and we discuss the results in section 5. finally, we conclude with possible directions for future work in section 6. ## related work most existing datasets for predicting the political ideology at the news article level were created by crawling the rss feeds of news websites with known political bias @xcite , and then projecting the bias label from a website to all articles crawled from it, which is a form of distant supervision. the crawling could be also done using text search apis rather than rss feeds @xcite . the media-level annotation of political leaning is typically obtained from specialized online platforms, such as news guard, foot_1 allsides, foot_2 and media bias/fact check, foot_3 where highly qualified journalists use carefully designed guidelines to make the judgments. as manual annotation at the article level is very time-consuming, requires domain expertise, and it could be also subjective, such annotations are rarely available at the article level. as a result, automating systems for political bias detection have opted for using distant supervision as an easy way to obtain large datasets, which are needed to train contemporary deep learning models. distant supervision is a popular technique for annotating datasets for related text classification tasks, such as detecting hyper-partisanship @xcite and propaganda/satire/hoaxes @xcite . for example, @xcite created a large corpus for detecting hyper-partisanship (i.e., articles with extreme left/right bias) consisting of 754,000 articles, annotated via distant supervision, and additional 1,273 manually annotated articles, part of which was used as a test set for the semeval-2019 task 4 on hyper-partisan news detection. the winning system was an ensemble of character-level cnns @xcite . interestingly, all topperforming systems in the task achieved their best results when training on the manually annotated articles only and ignoring the articles that were labeled using distant supervision, which illustrates the dangers of relying on distant supervision. @xcite extensively discussed the limitations of distant supervision in a text classification task about article-level propaganda detection, in a setup that is similar to what we deal with in this paper: the learning systems may learn to model the source of the article instead of solving the task they are actually trained for. indeed, they have shown that the error rate may drastically increase if such systems are tested on articles from sources that were never seen during training, and that this effect is positively correlated with the representation power of the learning model. they analyzed a number of representations and machine learning models, showing which ones tend to overfit more, but, unlike our work here, they fell short of recommending a practical solution. @xcite measured the bias at the article level using crowd-sourcing. this is risky as public awareness of media bias is limited @xcite . moreover, the annotation setup does not scale. finally, their dataset is not freely available, and their approach of randomly crawling articles does not ensure that topics and events are covered from different political perspectives. @xcite built a dataset annotated with the ideology of 594 articles related to the israeli-palestinian conflict published on bitterlemons. org. the articles were written by two editors and 200 guests, which minimizes the risk of modeling the author style. however, the dataset is too small to train modern deep learning approaches. @xcite built a dataset using distant supervision and labels from allsides. distant supervision is fine for the purpose of training, but they also used it for testing, which can be problematic. moreover, their training and test sets contain articles from the same media, and thus models could easily learn to predict the article's source rather than its bias. in their models, they used both the text and the url contents of the articles. overall, political bias has been studied at the level of news outlet @xcite @xcite , user @xcite , article @xcite , and sentence @xcite . in particular, @xcite developed a system to predict the political bias and the factuality of news media. in a followup work, @xcite showed that bias and factuality of reporting should be predicted jointly. a finer-grained analysis is performed in @xcite , where a model was trained on 10k sentences from a dataset of reviews @xcite , and used to discriminate objective versus non-objective sentences in news articles. @xcite presented a sentence-level classifier, where the labels were projected from the document level. ## dataset in this section, we describe the dataset that we created and that we used in our experiments. while most of the platforms that analyze the political leaning of news media provide in-depth analysis of particular aspects of the media, allsides stands out as it provides annotations of political ideology for individual articles, which ensures high-quality data for both training and testing, which is in contrast with distant supervision approaches used in most previous research, as we have seen above. in all-sides, these annotations are made as a result of a rigorous process that involves blind bias surveys, editorial reviews, third-party analysis, independent reviews, and community feedback. 5 furthermore, allsides uses the annotated articles to enable its balanced search, which shows news coverage on a given topic from media with different political bias. in other words, for each trending event or topic (e.g., impeachment or coronavirus pandemic), the platform pushes news articles from all sides of the political spectrum, as shown in figure 1 . we took advantage of this and downloaded all articles along with their political ideology annotations (left, center, or right), their assigned topic(s), the media in which they were published, their author(s), and their publication date. thus, our dataset contains articles that were manually selected and annotated, and that are representative of the real political scenery. note that the center class covers articles that are biased towards a centrist political ideology, and not articles that lack political bias (e.g., sports and technology), which commonly exist in news corpora that were built by scraping rss feeds. we collected a total of 34,737 articles published by 73 news media and covering 109 topics. 6 in this dataset, a total of 1,080 individual articles (3.11%) have a political ideology label that is different from their source's. this suggests that, while the distant supervision assumption generally holds, we would still find many articles that defy it. ## experiments and results we evaluated both the lstm and the bert models, assessing the impact of (i) de-biasing and (ii) incorporating media-level representation. ## conclusion and future work we have explored the task of predicting the leading political ideology of news articles. in particular, we created a new large dataset for this task, which features article-level annotations and is well-balanced across topics and media. we further proposed an adversarial media adaptation approach, as well as a special triplet loss in order to prevent modeling the source instead of the political bias in the news article, which is a common pitfall for approaches dealing with data that exhibit high correlation between the source of a news article and its class, as is the case with our task here. finally, our experimental results have shown very sizable improvements over using state-of-the-art pre-trained transformers. in future work, we plan to explore topic-level bias prediction as well as going beyond left-centerright bias. we further want to develop models that would be able to detect specific fragments in an article where the bias occurs, thus enabling explainability. last but not least, we plan to experiment with other languages, and to explore to what extent a model for one language is transferable to another one given that the left-center-right division is not universal and does not align perfectly across countries and cultures, even when staying within the western political world.