Concerns continue to grow about the prevalence of misinformation on social media platforms [25, 39] , including during the recent COVID-19 pandemic [40] . These types of content often exploit people's tendency to prefer pro-attitudinal information [20] , which can be exacerbated by platform content recommendations [5, 6] . In this paper, we explore a possible algorithmic approach to mitigate the spread of misinformation and promote content with higher journalistic standards online.
Social media platform recommendation algorithms frequently amplify bias in human consumption decisions. Though the information diets of Americans are less slanted in practice than many assume, the people who consume the most political news are most affected by the tendency toward selective exposure [15] . As a result, the news audience is far more polarized than the public as a whole [9, 17] . Although the prevalence of so-called "fake news" online is rather limited and concentrated among relatively narrow audiences [2, 3, [14] [15] [16] 18] , content that generally appeals to these tendencies -which include low-quality or false news -may generate high levels of readership or engagement [39] , prompting algorithms that seek to maximize engagement to distribute them more widely.
Prior research indicates that existing recommendation algorithms tend to promote items that have already achieved popularity [12, 30] . This bias may have several effects on the consumption of low-quality and false news. First, sorting the news by engagement can exacerbate polarization by increasing in-group bias and discouraging consumption among outgroup members [36] .
Second, it may contribute to information cascades, amplifying differences in rankings from small variations or random fluctuations and degrading the overall quality of information consumed by users [8, 11, 21, 26, 33] . Third, exposure to engagement metrics makes users more likely to share and less likely to fact-check highly engaging content from low-credibility sources, increasing vulnerability to misinformation [4] . Finally, popularity bias in recommendation systems can create socio-algorithmic vulnerabilities to threats such as automated amplifiers, which exploit algorithmic content rankings to spread low-quality and inflammatory content to like-minded audiences [34, 37] .
Given the speed and scale of social media, assessing directly the quality of every piece of content or the behavior of each user is infeasible. Online platforms are instead seeking to include signals about news quality in their content recommendation algorithms. A vast literature examines how to assess the credibility of online sources [7, 19] and the reputations of individual online users [1, 13] , which could in principle bypass the problem of checking each individual piece of content. Unfortunately, many of these methods are hard to scale to large groups and/or depend upon context-specific information about the type of content being generated (e.g., wikis). As a result, they are not easily applied to news content recommendations on social media platforms.
Another approach is to try to evaluate the quality of websites directly [43] , but scaling such an approach would likely be costly and cause lags in the evaluation of novel content. Similarly, while crowdsourced website evaluations have been shown to be generally reliable in distinguishing between high and low quality news sources [31] , such signals are vulnerable to manipulation as well as to delays in evaluating new sources.
Building on the literature about the benefits of diversity at the group level [22, 35] , we instead propose using the partisan diversity of the audience of a news source as a signal of its quality. This approach has two key advantages. First, audience partisan diversity can be computed at scale given that information about the partisanship of users is available or can be inferred in a reliable manner. Second, because diversity is a property of the audience and not of its level of engagement, it is less susceptible to manipulation if one can detect inauthentic partisan accounts [38, 41, 42] .
These two conditions (inferring partisanship reliably and preventing abuse by automated amplification/deception) could easily be met by the major social media platforms, which routinely collect a wealth of signals about their users and their authenticity.
We evaluate the merits of our proposed approach using data from two sources: a comprehensive data set of web traffic history from 6,890 Americans, collected along with surveys of self-reported partisan information from respondents in the YouGov Pulse survey panel, and a data set of 3,765 news source reliability scores of web domains compiled by trained experts in journalism and provided by NewsGuard [29] . We first establish that domain pageviews are not associated with overall news reliability, highlighting the potential problem with algorithmic recommendation systems that rely on popularity and related metrics of engagement. We next define measures of audience partisan diversity and show that these measures correlate with news reliability better than popularity does.
Finally, we study the effect of incorporating audience partisan diversity into algorithmic ranking decisions. When we create a variant of the standard collaborative filtering algorithm that explicitly takes audience partisan diversity into account, our new algorithm provides more trustworthy recommendations than the standard approach with only a small loss of relevance.
These results demonstrate that diversity in audience partisanship can serve as a useful signal of news reliability at the domain level, a finding that has important implications for the design of content recommendation algorithms used by online platforms. Although the news recommendation technologies deployed by platforms are more sophisticated than the approach tested here, our results highlight a fundamental weakness of algorithmic ranking methods that prioritize content that generates engagement and suggest a new metric that could help improve the reliability of the 4 FIG. 1. Relationship between audience size and news reliability by domain (Pearson r = 0.05, two-sided p = 0.12). Reliability scores provided by NewsGuard [29] .
recommendations that are provided to users.
To motivate our study, we first demonstrate that the popular news content that algorithmic recommendations often highlight is not necessarily reliable. To do so, we examine the relationship between audience size and news reliability in the YouGov Pulse data. Due to skew in audience size among domains, we use a logarithmic scale for the size. Fig. 1 shows that the amount of traffic that a website attracts is not associated with its news reliability, which we measure using NewsGuard scores (see Methods A). We do find a significant association if we consider websites with predominantly Democratic audiences (Pearson r = 0.08, two-sided p = 0.02) separately from those with predominantly Republican audiences (Pearson r = −0.06, two-sided p = 0.34), but the strength of association between the two variables is very weak overall (Pearson r = 0.05, two-sided p = 0.12). Domains for which we have NewsGuard reliability scores [29] are shaded in blue (where darker shades equal lower scores). Domains with no available score are plotted in gray.
In contrast, we observe that sites with greater audience partisan diversity tend to have higher NewsGuard scores while those with lower levels of diversity, and correspondingly more homogeneous partisan audiences, tend to have lower reliability scores. Fig. 2 shows how NewsGuard scores vary with both mean audience partisanship and variance in audience partisanship. The latter is our primary measure of audience partisan diversity at the website level (see Methods B).
As Fig. 2 indicates, unreliable websites with very low NewsGuard scores are concentrated in the tails of the distribution, where partisanship is most extreme and audience partisan diversity is, by necessity, very low. This relationship is not symmetrical: low-reliability websites (whose markers are darker shades of blue in the figure) are especially concentrated in the right tail, which corresponds to websites with largely Republican audiences. The data in Fig. 2 also suggests that the reliability of a website may be associated not just with the variance of the distribution of audience partisanship slants, but also with its mean. To account for this, we compute the coefficient of partial correlation between NewsGuard reliability scores and the variance of audience partisanship given the mean audience partisanship of each website. Compared with popularity, we find a stronger (and significant) correlation regardless of whether mean partisanship and audience partisan diversity are calculated by weighting individual audience members equally (user level, left panel: partial correlation r = 0.38, two-sided p < 10 −4 ) or by how often they visited a given site (pageview level, right panel: partial correlation r = 0.22, two-sided p < 10 −4 ). We study the diversity-reliability relationship in more detail in Fig. 3 , which differentiates between websites with audiences that are mostly Republican and those with audiences that are mostly Democratic. Consistent with what we report above, Fig. 3 shows that audience partisan diversity is positively associated with news reliability. Again, this relationship holds both when individual audience members are weighted equally (user level, left panel) and when they are weighted by their number of accesses (pageview level, right panel), though the association is stronger at the user level (standardized OLS coefficient: β = 0.28 (0.02) at user level; β = 0.17 (0.03) at pageview level). In addition, we find that the relationship is stronger for sites whose average visitor identifies as a Republican (standardized OLS coefficient of Republican domains: β = 0.41 (0.05) at user level; β = 0.30 (0.07) at pageview level) versus those whose average visitor identifies as a Democrat Of course, variance in audience partisanship is not the only possible way to define audience partisan diversity; alternative definitions can be used (e.g., entropy; see Methods B). As a robustness check, we therefore consider a range of alternative definition of audience partisan diversity and obtain results that are qualitatively similar to the ones presented here, though results are strongest for variance (see Supplementary Materials).
To understand the potential effects of incorporating audience partisan diversity into algorithmic recommendations, we next consider how recommendations from a standard user-based collaborative filtering (CF) algorithm [24, 32] change if we include audience partisan diversity as an additional signal. We call this modified version of the algorithm CF+D, which stands for Collaborative Filtering + Diversity (see Methods C for formal definition).
In classic CF, users are presented with recommendations drawn from a set of items (in this case, web domains) that have been "rated" highly by those other users whose tastes are most similar to theirs. Lacking explicit data about how a user would "rate" a given web domain, we use a quantity derived from the number of user pageviews to a domain (based on TF-IDF; see also Methods C) as the rating.
To evaluate our method, we follow a standard supervised learning workflow. We first divide web traffic data for each user in the YouGov Pulse panel into training and testing sets by domain (see Methods D). We then compute similarities in traffic patterns between users for all domains in the training set (not just news websites) and use the computed similarities to predict the aforementioned domain-level pageviews metric on the test set. The domains that receive the highest predicted ratings (i.e., expected TF-IDF-transformed pageviews) are then selected as recommendations.
Note that if a user has not visited a domain, then the number of visits for that domain will be zero. In general, due to the long tail in user interests [12] , we cannot infer that the user has a negative preference toward a website just because they have not visited it. The user may simply be unaware of the site. We therefore follow standard practice in the machine learning literature in only evaluating recommendations for content for which we have ratings (i.e., visits in the test set), though in practice actual newsfeed algorithms rank items from a broader set of inputs, which typically includes content the user may not have seen (for example, content shared by friends [5] ).
To produce recommendations for a given user, we consider all the domains visited by the user in the test set for which ratings are available from one or more respondents in a neighborhood of most similar users (domains with no neighborhood rating are discarded since neither CF nor CF+D can make a prediction for them, see Methods C) and for which we have a NewsGuard score (i.e., a reliability score). We then rank those domains by their rating computed using either CF or CF+D. This process produces a ranked list of news domains and reliability scores from both the standard CF algorithm and the modified CF+D algorithm, which incorporates the audience partisan diversity signal. We evaluate these lists using two different measures of trustworthiness, that are computed for the top k domains in each list: either the mean score (a number in the 0-100 range), or the proportion of domains with a score of 60 or higher, which NewsGuard classifies as indicating that a site "generally adheres to basic standards of credibility and transparency" [29] (see Methods E).
By varying the number of top domains k, we can evaluate how trustworthiness changes as the length of the list of recommendations increases. In Fig. 4 we plot the trustworthiness of the recommended domains as a function of k. We restrict values of k to 1-28, the values for which there are at least 100 users in each bin (plots spanning the full range are available in Supplementary Materials). Each panel compares the average trustworthiness of domains ranked by CF and CF+D with, as a baseline, the trustworthiness of websites users visited in the test set ranked by their TF-IDF-transformed number of visits (i.e., pageviews). This baseline captures the trustworthiness of the websites the users actually visited after adjusting for the fact that more popular websites tend to attract more visits in general.
We observe in Fig. 4 that the trustworthiness of recommendations produced by CF+D is significantly better than both standard CF recommendations and baseline statistics from user behavior. In particular, CF produces less trustworthy rankings than the user visit baseline (for small values of k the difference is within the margin of error), while CF+D produces rankings that are more trustworthy than both CF and the baseline across different levels of k. These results suggest that audience partisan diversity can provide a valuable signal to improve the reliability of algorithmic recommendations.
Of course, the above exercise would be meaningless if our proposed algorithm recommended websites that do not interest users. Because CF+D alters the set of recommended domains to prioritize those visited by more diverse partisan audiences, it may be suggesting sources that offer counter-attitudinal information or that users do not find relevant. In this sense, CF+D could represent an audience-based analogue of the topic diversification strategy from the recommender systems literature [44] . If so, a loss of predictive ability would be expected. Our results are generally encouraging. In both cases, precision is low and RMSE is high for low values of k, but error levels start to stabilize around k = 10, which suggests that making correct recommendations for shorter lists (i.e. k < 10) is more challenging than for longer ones. Moreover, comparing CF+D with CF, we see that though accuracy declines slightly for CF+D relative to CF, the difference is not statistically significant for all but small values of k, suggesting that CF+D is still capable of producing relevant recommendations.
Audience partisan diversity mitigates selective exposure to misinformation
The results above demonstrate that incorporating audience partisan diversity can increase the trustworthiness of recommended domains while still providing users with relevant recommendations. However, we know that exposure to unreliable news outlets varies dramatically across the population. For instance, exposure to untrustworthy content is highly concentrated among a narrow subset of highly active news consumers with heavily slanted information diets [14, 18] . We therefore take advantage of the survey and behavioral data available on participants in the Pulse panel to consider how CF+D effects vary by individual partisanship (self-reported via survey), behavioral measures such as volume of news consumption activity and information diet slant, and contextual factors that are relevant to algorithm performance such as similarity with other users.
In this section, we again produce recommendations using either CF or CF+D and measure their difference in trustworthiness with respect to a baseline based on user visits (specifically the ranking by TF-IDF-normalized number of visits v; see Methods C). However, we analyze the results differently than those reported above. Rather than considering recommendations for lists of varying length k, we create recommendations for different subgroups based on the factors of interest and compare how the effects of the CF+D approach vary between those groups.
To facilitate comparisons in performance between subgroups that do not depend on list length k, we define a new metric to summarize the overall trustworthiness of the ranked lists obtained with CF and CF+D over all possible values of k. Since users tend to pay less attention to items ranked lower in the list [23] , it is reasonable to assume that lower-ranked items ought to contribute less to the overall trustworthiness of a given ranking. Let us consider a universe of domains D as the set of items to rank. Inspired by prior approaches on stochastic processes based on ranking [10] , we consider a discounting method that posits that the probability of selecting domain d ∈ D from a given ranked recommendation list decays as a power law of its rank in the list:
where X ∈ D is a random variable denoting the probabilistic outcome of the selection from the ranked list, r d ∈ N is the rank of a generic d ∈ D, and α ≥ 0 is the exponent of power-law decay (when α = 0, all domains are equally likely; when α > 0, top-ranked domains are more likely to be selected).
Let us now consider probabilistic selections from two different rankings, represented by random variables X and X , where X is the random variable of the ranking produced by one of the two recommendation algorithms (either CF or CF+D) and X is the selection from the baseline ranking based on user visits. Using Eq. 1, we compute the expected change in trustworthiness Q from switching the selection from X to X,
where the expectations of Q(X) and Q(X ) are taken with regard to the respective rankings (see Applying Eq. 2, we find that CF+D substantially increases trustworthiness for users who tend to visit sources that lean conservative ( Fig. 6(a) ) and for those who have the most polarized information diets (in either direction; see Fig. 6 (c)), two segments of users who are especially likely to be exposed to unreliable information [2, 14, 18] . In both cases, CF+D achieves the greatest improvement among the groups where CF reduces the trustworthiness of recommendations the most, which highlights the pitfalls of algorithmic recommendations for vulnerable audiences and the benefits of prioritizing sources with diverse audiences in making recommendations to those users.
Note that even though the YouGov sample includes self-reported information on both party ID and partisanship of respondents, for stratification we use only the former (Fig. 6(b) ) but not the latter, to avoid circularity given the definition of CF+D, which does rely on the latter; instead, in Figs 6(a) and 6(c) we chose to stratify on an external measure of news diet slant (calculated from a large sample of social media users; see Methods H).
We also observe that CF+D has strong positive effects for users who identify as Republicans or lean Republican (Fig. 6(b) ) and for those who are the most active news consumers in terms of both total consumption (Fig. 6(d) ) and number of distinct sources (Fig. 6(e) ). Furthermore, since the two recommendation schemes considered here (CF and CF+D) are predicated on identifying similar users according to their tastes and behaviors, we also segment the users of the YouGov sample according to the degree of similarity with their nearest neighbors (identified based on Kendall's rank correlation coefficient between user vectors; see Methods C). Stratifying on the average of nearest neighbor similarities, we find that CF+D results in improvements for the users whose browsing behavior is most similar to others in their neighborhood and who might thus be most at risk of "echo chamber" effects ( Fig. 6(f) ). Finally, when we group users by the trustworthiness of the domains they visit, we find that the greatest improvements from the CF+D algorithm occur for users who are exposed to the least trustworthy information (Fig. 6(g) ). By contrast, the standard CF algorithm often recommends websites that are less trustworthy than those that respondents actually visit (∆Q < 0).
Our analysis combines two sources of data. The first is the NewsGuard News Website Reliability Index [29] , a list of web domain reliability ratings compiled by a team of professional journalists and news editors. The data that we licensed for research purposes includes scores of 3,765 web domains on a 100-point scale based on a number of journalistic criteria such as editorial responsibility, accountability, and financial transparency. 1 NewsGuard categorizes web domains into four main groups: "Green" domains, which have a score of 60 or more points and are considered reliable;
"Red" domains, which score less than 60 points and are considered unreliable; "Satire" domains, which should not be regarded as news sources regardless of their score; and "Platform" domains like Facebook or YouTube that primarily host content generated by users. The mean reliability score for domains in the data is 69.6; the distribution of scores is shown in Fig. 7 . Table I ).
A total of 6,890 participants provided data. In addition to their web traffic logs, participants reported their partisanship on a seven-point scale in online surveys. We perform a number of pre-processing steps on this data. We combine all waves into a single sample. We pool web traffic for each domain that received thirty or more unique visitors. Finally, we use the self-reported partisanship of the visitors to estimate mean audience partisanship and audience partisan diversity, which we estimate using different measures described next and evaluated in the Supplementary Materials.
To measure audience partisan diversity, first define N j as the count of participants who visited a web domain and reported their political affiliation to be equal to j for j = 1, . . . , 7 (where 1 = strong Democrat and 7 = strong Republican). The total number of participants who visited the domain is thus N = j N j , and the fraction of participants with a partisanship value of j is p j = N j /N . Denote the partisanship of the i-th individual as s i . We calculate the following metrics to measure audience partisan diversity: The above metrics all capture the idea that the partisan diversity of the audience of a web domain should be reflected in the distribution of its traffic across different partisan groups. Each weighs the contribution of each individual person who visits the domain equally; they can thus be regarded as user-level measures of audience partisan diversity. However, the volume and content of web browsing activity is highly heterogeneous across internet users [17, 27] , with different users recording different numbers of pageviews to the same website. To account for this imbalance, we also compute the weighted variants of the above audience partisan diversity metrics where, instead of treating all visitors equally, each individual visitor is weighted by the number of pageviews they made to any given domain.
As a robustness check, we compare the strength of association of each of these metrics to news reliability in the Supplementary Materials. We find that all variants correlate with news reliability, but the relationship is strongest for variance.
In general, a recommendation algorithm takes a set of users U and a set of items D and learns a function f : U × D → R that assigns a real value to each user-item pair (u, d) representing the interest of user u in item d. This value denotes the estimated rating that user u will give to item d. In the context of the present study, D is a set of news sources identified by their web domains (e.g., nytimes.com, wsj.com), so from now on we will refer to d ∈ D interchangeably as either a web domain or a generic item.
Collaborative filtering is a classic recommendation algorithm in which some ratings are provided as input and unknown ratings are predicted based on those known input ratings. In particular, the user-based CF algorithm, which we employ here, seeks to provide the best recommendations for users by learning from others with similar preferences. CF therefore requires a user-domain matrix where each entry is either known or needs to be predicted by the algorithm. Once the ratings are predicted, the algorithm creates a ranked list of domains for each user that are sorted in descending order by their predicted ratings.
To test the standard CF algorithm and our modified CF+D algorithm, we first construct a user-domain matrix V from the YouGov Pulse panel. The YouGov Pulse dataset does not provide user ratings of domains, so we instead count the number of times π u,d ∈ Z + a user u has visited a domain d (i.e., pageviews) and use this variable as a proxy [23] . Because this quantity is known to follow a very skewed distribution, we compute the rating as the TF-IDF of the pageview counts:
where π = u d π u,d is the total number of visits. Note that if a user has never visited a particular domain, then v u,d = 0. Therefore, if we arrange all the ratings into a user-domain matrix V ∈ R |U |×|D| , such that (V ) u,d = v u,d , we will obtain a sparse matrix. The goal of any recommendation task is to complete the user-domain matrix by predicting the missing ratings, which in turn allows us to recommend new web domains to users that they may not have seen. In this case, however, we lack data on completely unseen domains. To test the validity of our methods, we therefore follow the customary practice in machine learning of setting aside some data to be used purely for testing (see Methods D).
Having defined V , the next step of the algorithm is to estimate the similarity between each pair of users. To do so, we use either the Pearson correlation coefficient or the Kendall rank correlation of their user vectors; i.e., their corresponding row vectors in V (i.e., zeroes included). For example, if τ (·, ·) ∈ [−1, 1] denotes the Kendall rank correlation coefficient between two sets of observations, then the corresponding coefficient of similarity between u ∈ U and u ∈ U can be defined as:
where V u , V u ∈ R 1×|U | are the row vectors of u and u , respectively. A similar definition can be used for Pearson's correlation coefficient in place of τ .
These similarity coefficients are in turn used to calculate the predicted ratings. In the standard user-based CF, the predicted rating of a user u for a domain d is calculated as:
where N u d ⊆ U is the set of the n = 10 most similar users to u who have also rated d (i.e., the neighbors of u), v u ,d is the observed rating (computed with Eq. 3) that neighboring user u has given to domain d,v u andv u are the average ratings of u and u across all domains they visited, respectively, and sim(u, u ) is the similarity coefficient (computed with Eq. 4) between users u and u based on either the Pearson or the Kendall correlation coefficient.
Having defined the standard CF in Eq. 5, we now define our variant CF+D, which incorporates audience partisan diversity of domain d ∈ D as a re-ranking signal in the following way:
where g (δ d ) is the re-ranking term of domain d, obtained by plugging the audience partisan diversity δ d (for example, we use the variance of the distribution of self-reported partisan slants of its visitors, δ d = σ 2 d ) into a standard logistic function:
In Eq. 7, parameters a, ψ, and t generalize the upper asymptote, inverse growth rate, and location of the standard logistic function, respectively. For the results reported in this study we empirically estimate the location as t =δ, the average audience partisan diversity across all domains, which corresponds to the value ofδ = 4.25 since we measure diversity as the variance of the distribution of self-reported partisan slants. For the remaining parameters, we choose a = 1, ψ = 1. As a robustness check, we re-ran all analyses with a larger value of a and obtained qualitatively similar results (available upon reasonable request).
To evaluate both recommendation algorithms, we follow a standard supervised learning workflow. We use precision and root mean squared error (RMSE), two standard metrics used to measure the relevance and accuracy of predicted ratings in supervised learning settings. We define these two metrics elsewhere (see Methods F). Here, we instead describe the workflow we followed to evaluate the recommendation methods. Since our approach is based on supervision, we need to designate some of the user ratings (i.e., the number of visits to each domain, which are computed using Eq. 3) as ground truth to compute performance metrics.
For each user, we randomly split the domains they visited into a training set (70%) and a testing set (30%). This splitting varies by user: the same domain could be included in the training set of a user and in the testing set of another. Then, given any two users, their training set ratings are used to compute user-user similarities using Eq. 4 (which is based on Kendall's rank correlation coefficient; a similar formula can be defined using Pearson's correlation). If, in computing user-user similarities with Eq. 4, a domain is present for a user but not for the other, then the latter rating is assumed to be zero regardless of whether the domain is present in testing or not. This assumption, which follows standard practice in collaborative filtering algorithm, ensures that there is no leaking of information between the test and training sets.
Finally, using either Eq. 5 or Eq. 6, we predict ratings for domains in the test set and compare them with the TF-IDF of the actual visit counts in the data.
In addition to standard metrics of accuracy (Precision and RMSE; see Methods F), we define a new metric called trustworthiness to measure the news reliability of the recommended domains.
It is calculated using the NewsGuard scores in two ways: either using the numerical scores or the set of binary indicators for whether a site meets or exceeds the threshold score of 60 defined by NewsGuard as indicating that a site is generally trustworthy [29] . Let d 1 , d 2 , . . . , d k be a ranked list of domains. Using numerical scores, the trustworthiness is the average:
where Q(d) ∈ [0, 100] denotes the NewsGuard reliability score of d ∈ D.
If instead we use the binary indicator of trustworthiness provided by Newguard, then the trustworthiness of domains in a list is defined as the fraction of domains that meet or exceed the threshold score. Note that, unlike precision and RMSE, the trustworthiness of a list of recommendations does not use information on the actual ratings v u,d . Instead, using Eq. 8, we compute the trustworthiness of the domains in the test set ranked in decreasing order of user visits v u,d . We then compare the trustworthiness of the rankings obtained with either CF or CF+D against the trustworthiness of this baseline. 
Then the precision for a given value of k is given by the fraction of correctly predicted domains:
Similarly, the root mean squared error for a given value of k between the two ranked lists of ratings is computed as:
where ρ : [D] → D (respectively ρ ) is the inverse function of r(·) (respectively, r (·)); that is, the function that maps ranks back to their domain by the recommendation method (respectively, by actual visits). Note that, in the summation, ρ(r) and ρ (r) do not generally refer to the same web domain: the averaging is over the two ranked lists of ratings, not over the set of domains in common between the two lists.
To measure the effect of CF+D on the trustworthiness of rankings, we must select a particular list length k. Although Fig. 4 shows improvements for all values of k, one potential problem when stratifying on different groups of users is that the results could depend on the particular choice of k. To avoid dependence on k, we consider a probabilistic model of a hypothetical user visiting web domains from a ranked list of recommendations (Eq. 1) and define overall trustworthiness as the expected value of the trustworthiness of domains selected from that list (i.e., discounted by probability of selection). This procedure allows us to compute, for any given user, the effect of a recommendation method (either CF or CF+D) simply as the difference between its expected trustworthiness and the trustworthiness of the ranking obtained by sorting the domains visited by the user in decreasing order of pageviews (see Eq. 2).
In practice, to compute Eq. 2, let d 1 , d 2 , . . . , d k and d 1 , d 2 , . . . , d k be two ranked lists of domains, d r , d r ∈ D ∀r = 1, . . . , k, generated by a recommendation algorithm and by actual user pageviews, respectively, and let us denote with Q(d) the NewsGuard reliability score of d ∈ D (see Methods E).
Recall that Eq. 1 specifies the probability of selecting a given domain d ∈ D from a particular ranked list as a function of its rank. Even though any pair of equally-ranked domains will be different across these two lists (that is, d r = d r in general), their probability will be the same because Eq. 1 only depends on r. We can thus calculate the expected improvement in trustworthiness as:
where P (r) is the probability of selecting a domain with rank r from Eq. (1), which we computed setting α = 1. indicates domains with extreme liberal or conservative slants (i.e., they are almost exclusively shared by one group and not the other).
All authors designed the research. S.B. and S.Y. performed data analysis. All authors wrote, reviewed, and approved the manuscript. We repeat the analysis of Fig. 3 for all diversity metrics (see Methods B) and summarize the results in Table S1 . For each metric, we estimate the degree of linear association with news quality using the Pearson correlation coefficient. We also report the R 2 coefficient of determination and the two-sided p-value of the F-statistic as a measure of significance of the fit. Each metric is positively correlated with quality at the user level, but we find that the relationship is strongest for variance of audience partisanship. At the pageview level, however, the association disappears for all metrics but variance, which still produces a modest correlation. Finally, Figs. S6-S12 present results analogous to those in Fig. 6 , but using specific values of k. from Bakshy et al. [5] ) and by length of ranked list k. In this and the following plots, bars represent the standard error of the mean. Change in trustworthiness ∆Q based on scores from NewsGuard [29] .