What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Context (2020)

Abstract: Predicting the political bias and the factuality of reporting of entire news outlets are critical elements of media profiling, which is an understudied but an increasingly important research direction. The present level of proliferation of fake, biased, and propagandistic content online has made it impossible to fact-check every single suspicious claim, either manually or automatically. Thus, it has been proposed to profile entire news outlets and to look for those that are likely to publish fake or biased content. This makes it possible to detect likely “fake news” the moment they are published, by simply checking the reliability of their source. From a practical perspective, political bias and factuality of reporting have a linguistic aspect but also a social context. Here, we study the impact of both, namely (i) what was written (i.e., what was published by the target medium, and how it describes itself in Twitter) vs. (ii) who reads it (i.e., analyzing the target medium’s audience on social media). We further study (iii) what was written about the target medium (in Wikipedia). The evaluation results show that what was written matters most, and we further show that putting all information sources together yields huge improvements over the current state-of-the-art.

## introduction the rise of the web has made it possible for anybody to create a website or a blog and to become a news medium. undoubtedly, this was a hugely positive development as it elevated freedom of expression to a whole new level, thus allowing anybody to make their voice heard online. with the subsequent rise of social media, anybody could potentially reach out to a vast audience, something that until recently was only possible for major news outlets. one of the consequences was a trust crisis: with traditional news media stripped off their gate-keeping role, the society was left unprotected against potential manipulation. the issue became a general concern in 2016, a year marked by micro-targeted online disinformation and misinformation at an unprecedented scale, primarily in connection to brexit and the us presidential campaign. these developments gave rise to the term "fake news", which can be defined as "false, often sensational, information disseminated under the guise of news reporting." foot_0 it was declared word of the year 2016 by macquarie dictionary and of year 2017 by the collins english dictionary. in an attempt to solve the trust problem, several initiatives such as politifact, snopes, factcheck, and full fact, have been launched to fact-check suspicious claims manually. however, given the scale of the proliferation of false information online, it became clear that it was unfeasible to fact-check every single suspicious claim, even when this was done automatically, not only due to computational challenges but also due to timing. in order to factcheck a claim, be it manually or automatically, one often needs to verify the stance of mainstream media concerning that claim and the reaction of users on social media. accumulating this kind of evidence takes time, but time flies very fast, and any delay means more potential sharing of the malicious content on social media. a study has shown that for some very viral claims, more than 50% of the sharing happens within the first ten minutes after posting the micro-post on social media @xcite , and thus timing is of utmost importance. moreover, an extensive recent study has found that "fake news" spreads six times faster and reaches much farther than real news @xcite . a much more promising alternative is to focus on the source and to profile the medium that initially published the news article. the idea is that media that have published fake or biased content in the past are more likely to do so in the future. thus, profiling media in advance makes it possible to detect likely "fake news" the moment it is published, by simply checking the reliability of its source. from a practical perspective, political bias and factuality of reporting have not only a linguistic aspect but also a social context. here, we study the impact of both, namely (i) what was written (the text of the articles published by the target medium, the text and the audio signal in the videos of its youtube channel, as well as how the medium selfdescribes itself on twitter) vs. (ii) who read it (by analyzing the media readers in facebook, twitter, and youtube). we further study @xcite what was written about the target medium on wikipedia. our contributions can be summarized as follows: in the rest of this paper, we discuss some related work, followed by a description of our system's architecture and the information sources we use. then, we present the dataset, the experimental setup, and the evaluation results. finally, we conclude with possible directions for future work. ## related work while leveraging social information and temporal structure to predict the factuality of reporting of a news medium is not new @xcite @xcite @xcite , modeling this at the medium level is a mostly unexplored problem. a popular approach to predict the factuality of a medium is to check the general stance of that medium concerning already fact-checked claims @xcite @xcite . therefore, stance detection became an essential component in fact-checking systems @xcite . in political science, media profiling is essential for understanding media choice @xcite , voting behavior (dellavigna and kaplan, 2007), and polarization @xcite . the outlet-level bias is measured as a similarity of the language used in news media to political speeches of congressional republicans or democrats, also used to measure media slant @xcite . article-level bias was also measured via crowd-sourcing @xcite . nevertheless, public awareness of media bias is limited @xcite . political bias was traditionally used as a feature for fact verification @xcite . in terms of modeling, @xcite focused on predicting whether an article is biased or not. political bias prediction was explored by @xcite and @xcite , where news articles were modeled as left vs. right, or as hyperpartisan vs. mainstream. similarly, @xcite explored the left vs. right bias at the article level, modeling both textual and url contents of articles. in our earlier research @xcite , we analyzed both the political bias and the factuality of news media. we extracted features from several sources of information, including articles published by each medium, what is said about it on wikipedia, metadata from its twitter profile, in addition to some web features (url structure and traffic information). the experiments on the media bias/fact check (mbfc) dataset showed that combining features from these different sources of information was beneficial for the final classification. here, we expand this work by extracting new features from the existing sources of information, as well as by introducing new sources, mostly related to the social media context, thus achieving sizable improvements on the same dataset. in follow-up work @xcite , we showed that jointly predicting the political bias and the factuality is beneficial, compared to predicting each of them independently. we used the same sources of information as in @xcite , but the results were slightly lower. while here we focus on analyzing political bias and factuality separately, future work may analyze how the newly proposed features and sources affect the joint prediction. ## system and features in this section, we present our system. for each target medium, it extracts a variety of features to model (i) what was written by the medium, (ii) the audience of the medium on social media, and (iii) what was written about the medium in wikipedia. this results in multi-modal (text, speech, and metadata) feature set, which we use to train a classifier to predict the political bias and the factuality of reporting of news media. figure 1 illustrates the system architecture. ## conclusion and future work we have presented experiments in predicting the political ideology, i.e., left/center/right bias, and the factuality of reporting, i.e., high/mixed/low, of news media. we compared the textual content of what media publish vs. who read it on social media, i.e., on twitter, facebook, and youtube. we further modeled what was written about the target medium in wikipedia. we have combined a variety of information sources, many of which were not explored for at least one of the target tasks, e.g., youtube channels, political bias of the facebook audience, and information from the profiles of the media followers on twitter. we further modeled different modalities: text, metadata, and speech signal. the evaluation results have shown that while what was written matters most, the social media context is also important as it is complementary, and putting them all together yields sizable improvements over the state of the art. in future work, we plan to perform user profiling with respect to polarizing topics such as gun control @xcite , which can then be propagated from users to media @xcite . we further want to model the network structure, e.g., using graph embeddings @xcite . another research direction is to profile media based on their stance with respect to previously fact-checked claims @xcite , or by the proportion and type of propaganda techniques they use @xcite . finally, we plan to experiment with other languages.