Hidden Biases in Unreliable News Detection Datasets (2021)

Abstract: Automatic unreliable news detection is a research problem with great potential impact. Recently, several papers have shown promising results on large-scale news datasets with models that only use the article itself without resorting to any fact-checking mechanism or retrieving any supporting evidence. In this work, we take a closer look at these datasets. While they all provide valuable resources for future research, we observe a number of problems that may lead to results that do not generalize in more realistic settings. Specifically, we show that selection bias during data collection leads to undesired artifacts in the datasets. In addition, while most systems train and predict at the level of individual articles, overlapping article sources in the training and evaluation data can provide a strong confounding factor that models can exploit. In the presence of this confounding factor, the models can achieve good performance by directly memorizing the site-label mapping instead of modeling the real task of unreliable news detection. We observed a significant drop (>10%) in accuracy for all models tested in a clean split with no train/test source overlap. Using the observations and experimental results, we provide practical suggestions on how to create more reliable datasets for the unreliable news detection task. We suggest future dataset creation include a simple model as a difficulty/bias probe and future model development use a clean non-overlapping site and date split.

## introduction the proliferation of unreliable news is widely acknowledged @xcite @xcite , and its identification is a socially important problem. in this work we use the label unreliable news as a broad term for all unverifiable and misleading news content, regardless of whether the content is malicious (targeted misinformation) or not. accordingly, while specific definitions vary in different datasets used in this work, we refrain from using the term "fake" since identifying the intent of the author(s) is beyond the scope of this work. to mitigate the problem of surfacing unreliable news content, various websites (e.g., politifact foot_1 , media bias/fact check (mbfc) foot_2 , gossipcop foot_3 , etc.) determine the reliability of news by manually fact-checking the important claims in given news articles. beyond requiring investigative expertise, manual fact-checking is time-consuming and is thus limited to only a small set of selected news articles. recent research has explored automating this process using machine learning methods to automatically determine news veracity @xcite @xcite . these efforts were made possible due to the availability of large-scale unreliable news detection datasets @xcite @xcite . in our work, we examine if these datasets accurately reflect the real difficulty of this task or if there are any hidden biases in the datasets. specifically, we study different methods of dataset construction (e.g., how the data was collected, how the data was split, etc.) and show that the assessed difficulty of the task is sensitive to how carefully different factors are considered when building and using these datasets. our investigation begins with data collection procedures: we look at the source of news stories (news outlets, social media, fact-checking websites, etc.) as well as the annotation process (number of labels, granularity of labels, article-or site-level annotation). we discuss the pros and cons of each approach and point out some hidden pitfalls. using fakenewsnet @xcite as an example, we demonstrate how selection biases in data collection can lead to undesired biases in the created datasets. moving beyond data collection, we examine two commonly applied ways of splitting the dataset for training and testing that help the model achieve high performance without correctly modeling the task. specifically, we show that using a disjoint set of sites/news outlets for training and test data significantly decreases the models' performance (>10%) and that the drop in performance is related to how similar (or dissimilar) the sites in both sets are (reflected by various site-level distributional distance metrics including l2, cos, emd, etc.). additionally, we also examine the effect of time overlap between both train and test sets. we observe that different news outlets are likely to have similar content in a small time window (i.e., the same story gets covered by multiple outlets within a day or a few days period). while we do not find any evidence that the studied models exploit this factor, we nevertheless suggest that future datasets are split both by time and site/news outlet. in summary, our main contributions are: (1) showing how data collection procedures can lead to systematic biases in unreliable news datasets, (2) demonstrating how confounding factors--such as site/news outlet and time--in these datasets can degrade their quality and lead to underestimating the difficulty of the task, and finally (3) suggesting possible mechanisms to avoid these biases and confounding factors when building new datasets. to facilitate future research, we also provide a list of practical suggestions for data collection, dataset construction, and experiment design in dataset size article source label type nela 5 136k/713k/1.12m news outlets site-level fakenewsnet 6 603k fact-checking websites article-level r/fakeddit 7 1.06m social media (reddit) site-level pitfalls in data collection. datasets collected through crowd-sourcing or scraping the internet have the advantage of much better scalability compared to expert-annotated datasets. however, these automatic processes are prone to hidden pitfalls. gururangan et al. (2018); poliak et al. (2018) show that crowd-sourcing "natural language inference" datasets leads to various dataset biases. similar observations have been made for "fact verification" datasets @xcite . splitting data--for training, testing, and validation--is another important procedure in creating datasets that can lead to several problems. for example, @xcite show that models may just learn the patterns of certain annotators in a random split. @xcite demonstrated a significant overlap in current open-domain qa datasets. when present, these unexpected biases or overlaps in datasets can significantly undermine the utility of a dataset and lead to deceptively promising results that are in part due to artifacts of flaws in the dataset rather than successfully modeling the intended task. automated fact checking for statements. automated fact checking is an important task closely related to unreliable news detection, yet is constructed in a more controlled manner. this task focuses on strictly judging the factuality of one single statement instead of an entire article. @xcite first constructed a dataset with 106 claims from fact-checking websites with paired labels. fever @xcite is currently the largest scale fact-verification dataset, where 185,445 claims were generated by modifying sentences from wikipedia. both the altered claims and the ground truth supporting evidence are included in the dataset. existing effective approaches for fact-verification include self-attention based networks @xcite , large-scale pretrained transformers @xcite , neural retrieval methods @xcite , and reasoning 5 dataverse.harvard.edu/dataverse/nela 6 github.com/kaidmml/fakenewsnet 7 github.com/entitize/fakeddit on semantic-level graphs @xcite . ## unreliable news datasets collecting high-quality datasets plays an important role in automatic unreliable news detection research. here we review dataset collection strategies used in constructing recent datasets and point out some hidden pitfalls in these procedures. ## dataset split effect in this section, we study the effect of time and site/outlet overlap between the training and the evaluation set on the model's performance and show how these confounding factors can impact it. ## conclusion in this paper, we took a closer look at current largescale unreliable news detection datasets. we studied their collection procedures and dataset split strategies, and pointed out important flaws in the current approaches. specifically, we demonstrated that selection bias in dataset collection that often leads to undesired and significant artifacts in these datasets; highlighting confounding factors (e.g., article source, time) in news datasets that can lead to underestimating the difficulty of the task. finally we provide suggestions on how to better create and process such datasets in the future. we hope our work leads to more high-quality news datasets and that it inspires further work in this direction.