Reliability Estimation of News Media Sources: Birds of a Feather Flock Together (2024)

Abstract: Evaluating the reliability of news sources is a routine task for journalists and organizations committed to acquiring and disseminating accurate information.Recent research has shown that predicting sources’ reliability represents an important first-prior step in addressing additional challenges such as fake news detection and fact-checking.In this paper, we introduce a novel approach for source reliability estimation that leverages reinforcement learning strategies for estimating the reliability degree of news sources. Contrary to previous research, our proposed approach models the problem as the estimation of a reliability degree, and not a reliability label, based on how all the news media sources interact with each other on the Web.We validated the effectiveness of our method on a news media reliability dataset that is an order of magnitude larger than comparable existing datasets. Results show that the estimated reliability degrees strongly correlates with journalists-provided scores (Spearman=0.80) and can effectively predict reliability labels (macro-avg. F1 score=81.05).We release our implementation and dataset, aiming to provide a valuable resource for the NLP community working on information verification.

## introduction as of 2023, the number of internet users is over 5.18 billion worldwide, meaning that around twothirds of the global population is currently connected to the www (petrosyan, 2023). the web has democratized and radically changed how people consume and produce information by shifting the paradigm from a news-centred one to a usercentred one. nowadays, any person on the web can potentially be a "news medium" providing information either by creating websites, blogs and/or by making use of social media platforms. nevertheless, news media can no longer perform its role as "gatekeeper" deciding which stories to disseminate to the public or not (munger, 2020) since most of the information on the internet is unregulated by nature (cuan-baltazar et al., 2020). as a consequence, an enormous proliferation of misinformation has emerged leaving the public vulnerable to incorrect or misleading information about the state of the world which, among others, increased polarization and decreased trust in institutions and experts (lewandowsky et al., 2017; strömbäck et al., 2020). the world health organization (who) recently declared a worldwide "infodemic" characterized by an overabundance of misinformation (van der linden, 2022). the bestknown type of misinformation is fake news (lazer et al., 2018) defined as "false information intentionally created to mislead and/or manipulate a public through the appearance of a news format with an opportunistic structure to attract the reader's attention" (baptista and gradim, 2022). in an attempt to limit the impact of fake news, a large number of initiatives have been undertaken by media, journalists, governments, and international organizations to identify true and false information across the globe (shaar et al., 2020). for instance, the duke university's center for journalism research, the reporters' lab, lists a total of 419 fact-checking active sites online foot_0 from which factcheck.org, snopes, full fact and politifact are the most well-known. these sites manually and systematically assess the validity of thousands of claims. however, human annotators will always be outnumbered by the claims that need to be verified, reducing the impact of such services in a largescale scenario. consequently, we have witnessed a growing interest in using different machine learning models, ranging from non-neural (kwon et al., 2013; popat et al., 2016; nguyen et al., 2018) to deep learning-based ones (ma et al., 2016; wang, 2017; popat et al., 2018b; wang et al., 2018; fajcik et al., 2023), to determine the validity of claims, news and online information. nevertheless, these models' performance still has not reached confident accuracy values, limiting their applicability in real-world scenarios (baly et al., 2018). a more recent paradigm to fight fake news proposes to focus on the source rather than on the content (baly et al., 2018, 2020), a task referred to as profiling news medium. the underlying hypothesis states that even though fake news spreads mainly through social media, they still need an initial website hosting the news. hence, if information about websites is known in advance, identifying potentially fake news can be done by verifying the reliability of the source. in fact, this activity is also performed by journalists, who often consult rating services for news websites like newsguard foot_2 or mbfc foot_3 . nonetheless, these services are not exhaustive and difficult to keep up-to-date as they rely on human evaluators, highlighting the need for scalable automatic solutions that can be applied in real-world scenarios. previous research has shown that predicting reliability is an important first-prior step for factchecking systems (nguyen et al., 2018; popat et al., 2017; mukherjee and weikum, 2015) and also the most important aspect that journalists consider when manually verifying the trustworthiness of the information (baly et al., 2018). thus, in this paper, we focus on the task of source reliability estimation, i.e., automatically analyzing the source that produces a given piece of information and determining its reliability degree. concretely, we address the posed task by investigating the following research question: to what extent can we predict the reliability of a news media source solely based on its interactions with other sources? contrary to previous research, our proposed method represents a scalable and language-independent approach that can be further enriched via content-based features. our performed experiments shed light on the immediate (positive) effects of profiling news mediums through its interactions with other sources and also in combination with traditional content-based attributes. our research holds the potential to uncover deeper insights by incorporating more recent content-based technologies to further explore the nuanced dynamics of news websites, opening the door to a broader nlp research community. overall, the main contributions of this paper can be summarized as follows: (i) we propose a methodology capable of modeling the source reliability estimation problem in a real-world scale scenario that contrary to previous research, estimates the reliability degree (i.e. a continuous value) rather than a categorical value and does not depend on any third-party resources; (ii) we pioneer the introduction and evaluation of different algorithms to estimate the reliability score, exploring a spectrum from vanilla reinforcement learning strategies to task-specific variations; (iii) we build the largest news media reliability dataset available, orders of magnitude larger than existing datasets; (iv) we present empirical evidence that establishes the feasibility of predicting the reliability of a news media source solely through its interactions with other sources (which further improves when contentbased features are incorporated); and (v) we release both the dataset and source code to the wider nlp research community. foot_4 ## related work the task of determining information veracity has been approached from different angles and perspectives, from micro to macro, depending on the object of study. for instance, fact-checking focuses on validating a single statement, i.e. the claim; fake news detection analyses a whole document, i.e. the content of the news article. in this work, we focus on the source that produces a given piece of information, also known as source reliability estimation. within social media, the sources are individual users creating the content, and previous work has focused on identifying different types of users such as spammers (liubchenko et al., 2022; stringhini et al., 2010), bots (lei et al., 2023; knauth, 2019), fake profiles (roy and chahar, 2020; ramalingam and chinnaiah, 2018), paid users (mihaylov et al., 2015b), and trolls (tomaiuolo et al., 2020; miao et al., 2020; mihaylov et al., 2015a), among others (sansonetti et al., 2020; burdisso et al., 2022). however, in the broader case of the www, sources are individual websites (dong et al., 2015), and in our case, news media websites. previous studies have tangentially addressed news media source reliability as part of the study of automatic fact-checking systems, either as a prior in probabilistic graphical models (nguyen et al., 2018; mukherjee and weikum, 2015) or as features for stance classification models (popat et al., 2018a, 2017, 2016). in these studies, reliability estimation relied on indirect measures since no gold labels were used. for instance, some works use the alexarank foot_5 and pagerank (brin and page, 1998) scores of the websites as proxies for their reliability (baly et al., 2018; popat et al., 2016) while others the proportion of articles that refute false claims and support true claims (popat et al., 2018a, 2017). however, in the latter, authors rely on a fact-checking model and the selected true and false claims while, in the former, on scores that only capture the authority and popularity of the sources, not necessarily their trustworthiness -for instance, think of popular unreliable gossip websites foot_6 or satirical news websites, like the onion, foot_7 highly popular, attracting huge web traffic. recently, baly et al. (2020, 2019, 2018) addressed the source reliability estimation task on its own, modeling it as a classification task using source-level gold annotations. in particular, authors focused on predicting websites factual reporting and political bias using the values published by a news media rating service as ground truth. however, their proposed method relies on collecting and extracting information from multiple external and restricted sources (e.g. twitter, facebook, youtube, etc.) for generating content-based, audience-based, and metadata-based features for the classification model, limiting its practical use on a large-scale scenario. in this paper, we also address the task using gold annotations, however, we adopt an easier-to-scale approach. specifically, we model the problem as estimating a continuous value (i.e. the reliability degree) based simply on how all news media sources interact with each other on the world wide web. ## experiments and evaluation results for experimentation, we define the reward values based on ground truth labels as r(s) = 1 if the label is "reliable", r(s) = -1 if "unreliable", and r(s) = 0 otherwise. in addition, selecting appropriate hyperparameter values is crucial. for i-reliability, n controls how far to look in the neighborhood for investments (how many nodes away). similarly, in the reinforcement learning strategies, the discount factor γ controls the distance of looking back/forward; γ ≈ 0 focuses mostly on present reward r(s), while γ ≈ 1 considers all history/future to compute ρ(s). we performed a grid search over @xmath0, 10] and γ ∈ [0.05, 0.95] to determine the best hyperparameter values on each of the three experimental sets. the grid search was performed using 5-fold cross validation selecting the n and γ that obtained the best macro avg. f 1 on the reliability classification task, as described in section 5.1. we observed that, independently of the dataset, better reliability estimation is achieved when looking mostly at nearby sources, as better performance was obtained with small n (@xmath1.5) values -details in appendix a. ## conclusion and future work in this study, we introduced an approach for assessing the reliability of news media through their network interactions. this approach diverges from previous models that depend on content, audience feedback, and/or metadata. moreover, unlike in previous works, our method estimates a reliability degree rather than a reliability label. we assessed the quality of the estimated values in terms of reliability classification and correlation with journalistsprovided scores. we found that a source's origins is more indicative of its reliability than its reach and show evidence that it is feasible to predict the reliability of news media using only their network interactions, providing an easier-to-scale approach than prior methods. as future work, we plan to expand the study by building a larger graph and designing more sophisticated strategies that leverage content-based features. additionally, we aim to explore the estimation of other news source properties, such as political bias, using the same approach. finally, we intend to investigate the use of the estimated reliability values in downstream tasks like fact-checking and fake news detection. @xcite in fact, we performed an additional experiment in which we set ρ(s) = ρp(s)+ρ f p (s) 2 , i.e., the reliability degree was defined as the average of the values obtained by the best performing strategies, p-reliability and fp-reliability, obtaining the strongest correlation values (@xmath2.933, @xmath3.803 and @xmath4.715, @xmath5.697 with and without rewards, respectively). @xcite in contrast to reliability classification, where ρ(s) = 0 is the fixed threshold separating reliable from unreliable sources. @xcite we are releasing the list of estimations for all 17k sources along with this paper.