Predicting the Factuality of Reporting of News Media Using Observations about User Attention in Their Y ou T ube Channels (2021)

Abstract: We propose a novel framework for predicting the factuality of reporting of news media outlets by studying the user attention cycles in their YouTube channels. In particular, we design a rich set of features derived from the temporal evolution of the number of views, likes, dislikes, and comments for a video, which we then aggregate to the channel level. We develop and release a dataset for the task, containing observations of user attention on YouTube channels for 489 news media. Our experiments demonstrate both complementarity and sizable improvements over state-of-the-art textual representations.

## introduction disinformation in the news and in social media is perceived as having a major impact on society, e.g., during the 2016 us presidential election @xcite and the brexit referendum @xcite . during the covid-19 pandemic outbreak, the moral panic @xcite around online disinformation grew to a whole new level as the first global infodemic. 1 nowadays, fighting disinformation online has been recognized as one of the most important issues societies around the world are facing today. in this paper, we highlight an aspect of disinformation that is often neglected. rather than examining the truth-value of individual piece of information, we investigate the general quality of the attention regimes in different news outlets, by analyzing news media's youtube channels. ## mit technology review: tinyurl.com/y8oschng youtube is the largest and the most popular platform for video sharing with over two billion users and it is also the second most widely used news source in usa, after facebook. 2 while the platform has been scrutinized for the way in which it may amplify marginal and sometimes radical contents @xcite , the connection between attention dynamics and disinformation levels is still largely unexplored. here, we do not focus on specific videos, but rather on entire youtube channels of news media and their attention dynamics. in particular, we are interested in differentiating the "attention cycles" @xcite of youtube channels, that is to assess the rapidity and the steepness with which their videos rise and fall in the consideration of their audiences. while some outlets encourage extensive and diverse discussions, other tend to concentrate everyone's attention on the latest hot-button, thus distracting the public opinion instead of nourishing it @xcite . our contributions are the following: the paper is organized as follows: section 2 presents related work. section 3 describes our dataset. section 4 discusses our methodology. section 5 presents the experiments and results. section 6 offers analysis and discussion. section 7 concludes and points to directions for future work. ## related work significant efforts have been dedicated in the last years to automating the detection of disinformation (which is commonly referred to as fake news), which we look at more closely in this section. at the end, we mention previous attempts to use data from youtube for media classification tasks. analysis of the content many approaches have been proposed to analyze both the style and the content of fake news. using natural language processing techniques, @xcite pointed out how fake news can be characterized by stylistic features such as the overuse of proper nouns, punctuation, capital letters, negation terms, and repetitions in the text @xcite . fake news has also been associated with intensity of sentiment and emotions, compared to mainstream news @xcite . here, we also use textual representations, but (i) we focus mainly on analyzing the user attention cycles in youtube channels, and (ii) we aim at categorizing entire news media outlets rather than individual pieces of news. ## data we started from a corpus of news media outlets, whose reliability has been evaluated by media bias/fact check foot_2 (mbfc). lead by a team of independent journalists and researchers, mbfc has analyzed close to 4,000 news outlets over the past six years. for each news outlet, they provide a detailed analysis summarized by a 'factuality' score chosen among: very high, high, mostly factual, mixed, low, and very low. we searched the youtube channels of news outlets in the mbfc corpus and we monitored all the videos they published from february'2020 to august'2020. using the youtube data api, 5 we collected the number of views, likes, dislikes and comments collected during the first seven days after the publication of each video. we also stored its title and its description. we observed that the percentage of media channels labelled with the very high and the very low categories was 3.1% and 1%, respectively. we thus merged very high with high; mostly factual with mixed; and very low with low, ending up with a 3-way labelling: high, mixed, and low. finally, for the sake of data balancing, we excluded the channels with fewer than 20 videos, and we capped the most prolific channels at the newest 100 videos. the final distribution of channels and their corresponding videos for each level of factuality is shown in ## method our system is composed of two main components focusing on (i) data preparation, and on (ii) sequential classification, respectively. below, we describe the representation we use for video-level and also for channel-level classification. ## experiments and evaluation we train two subsequent classifiers for factuality prediction: for videos and for channels. we conduct experiments with different models and we compare them to a majority-class baseline. we evaluate the models in terms of accuracy, balanced accuracy, and mean absolute error (mae). mae is a more relevant measure in our case as it takes into account the ordering of the labels: confusing high factuality with mixed factuality is a smaller error than confusing it with low factuality. while our ultimate goal is to classify channels, we start with video classification, then we aggregate the predictions, and we use them to make predictions at the channel level. we divide the dataset into training, development, and test split at the channel level. then, for the video-level experiments, we use for training/development/testing the videos for the respective channels. note that this guarantees that all videos for a given channel go into the same split. ## discussion below, we analyze the results and we perform an ablation study. ## conclusion and future work we proposed a novel framework for predicting the factuality of reporting of news outlets by studying the user attention cycles in their respective youtube channels. we further designed a rich set of features derived from the temporal evolution of the number of views, likes, dislikes, and comments for a video, which we then aggregated at the channel level. our experiments demonstrated both complementarity and sizable improvements over state-of-the-art textual representations. we further developed and released a dataset containing observations about user attention on youtube channels for 489 news media. we hope that this will enable future research on using data from video sharing platforms. in future work, we plan to improve the class imbalance of the dataset by extending it with more examples. we further want to integrate additional features based on the comments for the videos and other information sources such as twitter and wikipedia. finally, we plan to study the utility of user attention cycles for other related tasks such as political ideology detection for news media. ethics and broader impact data collection our dataset was collected from youtube, using their public api. user privacy our dataset contains aggregated attention statistics without any user data. biases any biases found in the dataset are unintentional, and we do not intend to do harm to any group or individual. intended use and misuse potential our dataset and the proposed model can enable the development of systems for automatic detection of reliable/unreliable youtube channels, which could support media literacy, as well as analysis and decision making for the public good. however, they could also be misused by malicious actors. environmental impact. finally, we would also like to warn that the use of large-scale transformers requires a lot of computations and the use of gpus/tpus for training, which contributes to global warming @xcite .