A Survey on Predicting the Factuality and the Bias of News Media (2024)

Abstract: The present level of proliferation of fake, biased, and propagandistic content online has made it impossible to fact-check every single suspicious claim or article, either manually or automatically. An increasing number of scholars are focusing on a coarser granularity, aiming to profile entire news outlets, which allows fast identification of potential “fake news” by checking the reliability of their source. Source factuality is also an important element of systems for automatic fact-checking and “fake news” detection, as they need to assess the reliability of the evidence they retrieve online. Political bias detection, which in the Western political landscape is about predicting left-center-right bias, is an equally important topic, which has experienced a similar shift toward profiling entire news outlets. Moreover, there is a clear connection between the two, as highly biased media are less likely to be factual; yet, the two problems have been addressed separately. In this survey, we review the state of the art on media profiling for factuality and bias, arguing for the need to model them jointly. We also shed light on some of the major challenges for modeling bias and factuality jointly. We further discuss interesting recent advances in using different information sources and modalities, which go beyond the text of the articles the target news outlet has published. Finally, we discuss current challenges and outline future research directions.

## introduction the rise of the web has made it possible for anybody to create a website and to become a news medium. this was a hugely positive development as it elevated freedom of expression to a whole new level, allowing anybody to have their voice heard. with the subsequent rise of social media, anybody could potentially reach out to a vast audience, something that until recently was only possible for major news outlets. one of the consequences was a trust crisis: with traditional news media stripped of their gatekeeping role, society was left unprotected against potential manipulation. in an attempt to solve the trust problem, several initiatives, such as politifact foot_0 , snopes foot_1 , factcheck foot_2 , and full fact foot_3 , have been launched to fact-check suspicious claims manually. however, given the scale of the proliferation of false information online, it was unfeasible to fact-check every single suspicious claim, even when this was done automatically @xcite , not only for computational reasons but also due to timing. in order to fact-check a claim manually or automatically, it is required to verify the stance of mainstream media concerning that claim and/or the reaction of users on social media. accumulating this evidence takes time, and delay means more potential sharing of the malicious content @xcite . therefore, a much more promising alternative is to profile the medium that initially published the news article with the suspicious claim. since media that have published fake or biased content in the past are more likely to do so in the future, profiling media in advance makes it possible to detect likely "fake news" the moment it is published by simply checking the reliability of its source @xcite @xcite . factuality labels at the media level can also be used for distant supervision, labeling all their articles with the medium's label. this approach is frequently used for "fake news" detection where manually annotating large datasets can be challenging @xcite . estimating news source reliability is important for claim fact-checking @xcite , and it also gives an important prior when solving article-level tasks such as "fake news" and click-bait detection @xcite @xcite @xcite @xcite . moreover, the clef-2023 checkthat! lab had a recent shared task on predicting the factuality and the bias of news media @xcite @xcite . recently, @xcite conducted a comprehensive examination of research methodologies, exploring the influence of media framing on public opinions, political dynamics, and policies. they stress the importance of understanding media bias and framing to enhance media literacy, critical thinking, and informed decision-making. several surveys on fake news @xcite @xcite @xcite , mis/dis-information @xcite @xcite , fact-checking @xcite @xcite , truth discovery @xcite @xcite , and propaganda detection @xcite have been conducted. however, they have focused on claims or articles, while here we survey research on profiling entire news outlets. we delve into the capacity of large language models (llms) to estimate the credibility and political leaning of sources through carefully crafted questions. we also provide a holistic perspective on the use of llms in the context of media profiling. ## factuality veracity of information has been studied at different levels: (i) claim-level (e.g., fact-checking), (ii) article-level (e.g., "fake news" detection), (iii) user-level (e.g., hunting for trolls), and (iv) medium-level (e.g., source reliability estimation). our primary interest here is in the latter. at the claim-level, significant effort has been paid to fact-checking and rumor detection using information from social media, i.e., how users reply to the claim @xcite @xcite @xcite @xcite @xcite @xcite ), yet, the need for more nuanced methodologies that concentrate on a multitude of characteristics is paramount @xcite . a set of web pages and snippets from search engines have also been used as a source of information @xcite @xcite @xcite . in either case, the most important information for the claim-level tasks are stance (does a tweet or a news article agree or disagree with the claim?) and source reliability (do we trust the user who posted the tweet or the medium that published the news article?). the problem of source reliability remains largely under-explored. in the case of social media and community fora, it concerns modeling the user, e.g., there has been research on finding opinion manipulation trolls @xcite , sockpuppets @xcite , internet water army @xcite , and seminar users @xcite . in the case of the web, it is about source trustworthiness (the url domain, the medium). in early work, the source reliability of news media has often been estimated automatically based on the general stance of the target medium with respect to known true/false claims without access to gold labels about the overall medium-level factuality of reporting @xcite @xcite @xcite . more recent work has addressed the task as one on its own right. @xcite used gold labels from media bias/fact check (mbfc) foot_4 , and rich information sources available before disinformation campaign begins: articles published by the medium, what is said about it on wikipedia, metadata from its twitter profile, url structure, and traffic information to characterize the media. in follow-up work, @xcite uncovered the ordinal relationship between media bias and factuality using a multi-task ordinal regression setup, as detailed in section 4. then, @xcite considered the social context, extended the information sources to include facebook followers and speech signals from the news medium's channel on youtube (if any). @xcite focused on infrastructure features such as domain registrations, tls/ssl certificates, and web hosting configurations. they posit that these features could potentially reveal significant disparities between disinformation and authentic news websites, prior to content dissemination. @xcite leveraged the concept of communities to connect users, the content they prefer, and with the source which provide that content. they approached the task as a problem of reasoning over relationships between sources, their published articles, and the engagement patterns of users on social media within a graph framework. @xcite proposed a graph-based framework to profile news media outlets, with nodes representing the outlets and the connecting edges indicating audience overlap. taking inspiration from homophily considerations, they constructed a network reflecting the hypothesis that similar users consume similar types of media. large language models (llms) can also be used to estimate source reliability as they can capture knowledge @xcite . @xcite showed that with well-crafted instructions, chatgpt showcases its capability to provide credibility ratings for an extensive spectrum of news outlets. their results showed that these ratings correlate with those from human experts and that llms could be an affordable reference for credibility ratings in media profiling applications. @xcite introduced an innovative interactive framework for news media profiling, combining the capabilities of graph-based models, pre-trained llms, and human expertise to delve into the social context of news on social media platforms. their findings highlighted the framework's ability to rapidly identify fake news media, even in the challenging terrain of emerging news events, with as few as five human interactions. ## bias compared to factuality, which can be objectively determined by whether a piece of information is true or not, media bias has more complex dimensions. for the last few decades, many scholars have conceptualized media bias in different ways. for instance, a bias can be defined as "imbalance or inequality of coverage rather than as a departure from truth" @xcite . a departure from truth, however, can be measured only when an accurate record of the event is available (e.g., trial transcript and reporting). a different definition, namely "any systematic slant favoring one candidate or ideology over another" @xcite , is proposed to capture various dimensions rather than coverage imbalance, such as favorability conveyed in visual representations (i.e., news photos). e.g., smiling, speaking at the podium, cheering crowd, and eye-level shots are preferred over frowning, sitting, being alone, and shots from above, respectively. @xcite utilized pre-trained bert @xcite models, fine-tuned on news articles from various media outlets, to capture linguistic biases through masked language modeling. to validate their model's ability to detect media bias, they compare its results with established news bias datasets from sources like pew research and allsides.com d'alessio and @xcite reviewed 59 studies about partisan media bias in presidential elections. they proposed to categorize media bias into the following three types: (i) gatekeeping bias, where editors and journalists "determine" which content reaches the audience within various forms of media @xcite , (ii) coverage bias, where the amount of news coverage (e.g., the length of newspapers articles, or the time given on television) each party receives is systematically biased to one party at the expense of the other one @xcite , and (iii) statement bias, where news media interject their attitudes or opinions in the news reporting. @xcite proposed a more relaxed concept of media bias, which is "a portrayal of reality that is significantly and systematically (not randomly) distorted," to take a variety of media bias dimensions into account. in particular, he focused on two main forms of media bias-selection bias (i.e., what to cover) and presentation bias (i.e., how to cover it)-driven by the choices of newsmakers. selection bias has been studied in various ways, including qualitative interviews or surveys of journalists and editors about the decision-making process they use to select the stories in their newsroom (tandoc jr, 2014). here, news selection is not necessarily confined to the political context. news reporting about any news items can be considered as a unit of analysis. data-driven research on selection bias commonly follows three steps: (i) collect news articles (for newspapers or online news) or transcripts (for tv news) for a target period, (ii) conduct content analysis to find the news coverage of politicians, parties, or events. optionally, study the tone of the news articles (e.g., negative news are more frequently reported) @xcite , and (iii) identify systematic biases by comparing news coverage. an exhaustive database of news stories is thus essen-tial for selection bias research. while commercial databases, such as lexis nexis, have been widely used @xcite @xcite , publicly available datasets, such as gdelt, start to get attention @xcite and are getting validated by comparing multiple sources @xcite . the availability of such datasets also enables researchers to compare news coverage across countries @xcite @xcite . presentation bias has been characterized from diverse perspectives, including framing @xcite , visuals @xcite , sources @xcite , tone (soroka, 2012), and more. particularly, framing bias has been actively studied in many disciplines. framing bias refers to a bias that highlights a certain aspect of an event or an issue more than the others @xcite . emphasizing a particular aspect can deliver a distorted view toward the issue even without the use of biased expressions. framing biases have been typically studied at issue level @xcite . researchers collect news articles about an issue or an event, conduct manual content analysis, and build a frame detection model @xcite . open-source tools to help the analysis have been proposed @xcite . while this approach can characterizes diverse frames, it is not trivial to compare framing across issues. the media frames corpus (mfc) was proposed to address this limitation @xcite . it contains articles annotated with 15 generic frames (including others) across three policy issues. several studies have demonstrated reasonable prediction performance of the general media frames with different datasets @xcite . these 15 general frames were also used for analyzing political discourse on social media @xcite . these frames are often customized to a specific issue by adding issue-specific frames @xcite , even though doing so somewhat contradicts the original motivation of general media frames, namely to be able to compare frames across various issues. news slant was proposed to characterize how framing in news reports favors one side over the other @xcite . the media-level slant thus could differ across issues @xcite . a variety of methods have been proposed to quantify the extent of news slant in traditional news media by (i) linking media outlets to politicians with known political positions, (ii) directly analyzing news content, and (iii) using shared audience among media outlets. @xcite assigned an ada (americans for democratic action) score for each media outlet by investigating co-citations of think-tanks by members of congress and media outlets. @xcite proposed an ideological slant index of news media in a seminal study. the news slant is measured by the extent of phrases in news coverage that are more frequently used by one political party (i.e., democratic or republican) congress members than by another one in the 2005 congress record. their frequency-based approach successfully finds politically charged phrases such as death tax or war on terror by republicans and associated media and estate tax or war in iraq by democrats and associated media, and they further computed media slant index for 433 newspapers. the choice of words by political party members and news media is considered framing because they purposely highlight some aspects of the issue over others. @xcite proposed a method to compute media slant scores by measuring distances between media sources by their mutual followers on twitter. @xcite identified the political leanings of media outlets and influential people on twitter based on their stance on controversial topics. they built clusters of users around core vocal ones based on their behavior on twitter, such as retweeting, using a procedure proposed in @xcite . left-center-right bias (or left-right bias) was studied based on media-level annotation from specialized online platforms, such as news guard, allsides, and media bias/fact check, where journalists use carefully designed guidelines to make judgments. researchers have then trained systems to predict this bias using a variety of information sources such as analyzing the corresponding youtube channels @xcite , and using information from the articles the target news outlet has published, what is there about them in social media and in wikipedia @xcite . there has also been work on predicting the leftcenter-right bias of articles, which is somewhat relevant here as it can be an element of media-level analysis. such systems are typically trained using distant supervision, projecting the label from a medium to each article from that medium, which is an easy way to obtain large datasets, needed to train contemporary deep learning models. for example, @xcite used site-level annotations from the allsides website for political bias detection. the same approach was used to study hyperpartisanship, i.e., extremely one-sided reporting @xcite , as part semeval-2019 task 4 on hyper-partisan news detection @xcite . recent studies have underscored the potential problems of inaccuracy and noise linked to distant supervision. this has led to the development of a left-center-right bias dataset, which features careful manual annotations at the article level @xcite . ## joint modeling figure 1 shows the correlation between the factuality and the bias of the news outlets analyzed by mbfc. we can see that extreme-left and extreme-right websites are not factual, and that center media are most factual overall foot_5 . moreover, some of the datasets used for the two tasks have media-level annotations for both factuality and bias. thus, it makes sense to model factuality and bias jointly. yet, joint modeling of the two tasks remains severely underexplored. in fact, there has been a single attempt at doing so to date: @xcite proposed a multi-task learning formulation based on the copula ordinal regression frame-work @xcite , which jointly predicts factuality and bias on ordinal scales. they further used several auxiliary tasks, modeling centrality, hyper-partisanship, as well as left-vs.-right bias on a coarse-grained scale. they further took into account the ordinal nature of the labels for both factuality (high, mix, low) and bias (ext. left, left, center left, center, center right, right, ext. right) tasks, noting that classifying an extreme right medium as extreme-left is a huge error, while classifying it as a center is a smaller one, and predicting right is an even smaller error. the chart foot_6 where reuters foot_7 and airforcetimes foot_8 are center-bias and exhibit high factuality. ## major challenges ordinal scales: while ideological bias (news slant) is typically modeled as left-center-right, a spectrum can exist within each bias based on bias intensity. a hyperpartisan bias prediction task has been tested to differentiate far-right from right and far-left from left, but it does not model the political bias using an ordinal scale. difficulties in labeling the bias (i.e., creating ground-truth datasets) by experts or through crowdsourcing presents a major hurdle for modeling ideological bias as an ordinal variable. multimodality: in news reporting, photos typically attract high attention, and readers can sometimes understand a news story from the photos alone, even without reading the text. indeed, news text and photos are strongly coupled and deliver relevant information about news stories to readers. thus, modeling news text and photos together should benefit our understanding of their factuality @xcite and potential harmfulness @xcite . ## lessons learned factuality and bias have some commonalities as they exert negative influences on the public by delivering skewed information. political landscape positioning: news media often take a biased position in the political landscape to appeal to partisan audiences, particularly as the news industry becomes more competitive. reporting perspective: despite the bias, many journalists and editors are concerned about the issue and try to report diverse perspectives. multi-dimensionality: media bias is multidimensional and can be conveyed through different means such as text, photos, and videos, with ideological bias being a significant conceptualization. factuality challenges: accurate prediction of ideological bias of a news medium is easier than assessing factuality, as factuality relies on verification from other sources and observations. sophisticated analysis: more sophisticated analysis of text style and multimedia characteristics could improve accuracy, but other elements of a news medium need to be complemented. audience homophily: recent studies emphasize audience homophily and infrastructure characteristics in bridging performance gaps, valuable for early discovery and categorization of news media due to their content-agnostic nature. ## conclusion we surveyed the emerging field of news media profiling for factuality and bias, which can enable early fake news detection. while this is a relatively understudied direction, there has been a lot of recent interest in the problem, including a shared task at the clef-2023 checkthat! lab. we discussed that the factuality and the bias of news outlets are a matter of degree, and we advocated for their joint modeling. besides textual representations, we examined the contribution of contextual frameworks. finally, we discussed the major challenges and made a forecast for the future.