Mitigation of Diachronic Bias in Fake News Detection Dataset (2021)

Abstract: Fake news causes significant damage to society. To deal with these fake news, several studies on building detection models and arranging datasets have been conducted. Most of the fake news datasets depend on a specific time period. Consequently, the detection models trained on such a dataset have difficulty detecting novel fake news generated by political changes and social changes; they may possibly result in biased output from the input, including specific person names and organizational names. We refer to this problem as Diachronic Bias because it is caused by the creation date of news in each dataset. In this study, we confirm the bias, especially proper nouns including person names, from the deviation of phrase appearances in each dataset. Based on these findings, we propose masking methods using Wikidata to mitigate the influence of person names and validate whether they make fake news detection models robust through experiments with in-domain and out-of-domain data.

## introduction fake news, which refers to intentional and verifiable false news stories, has caused significant damage to society. for example, @xcite noted that 25% of the news stories linked in tweets posted just before the 2016 u.s. presidential election were either fake or extraordinarily biased. in addition to elections, fake news tends to spread after unusual situations such as natural disasters @xcite and disease outbreaks (e.g., @xcite ). to address these problems, various studies on the development of models for fake news detection from social media posts and news content, and the construction of fake news datasets for this purpose have been conducted @xcite . most datasets for fake news detection consist of factual and fake news that actually diffuse over the internet. the topics and contents of fake news change over time because they are strongly influenced by the interests of the general population @xcite . for instance, there were fake news related to president obama in 2013 @xcite , presidential election in 2016 @xcite , and covid-19 in 2020 @xcite . thus, datasets are frequently constructed based on fake news present in a specific period. fake news detection models learned from these datasets achieve high accuracy for the datasets constructed for the same period and domain, while they lead to a drop in the detection performance of fake news in different domains and future applications because of the difference in word appearance. in other words, fake news detection models learned from a dataset that includes news only from a specific period may possibly result in a biased judgment from the input, based on cues related to a particular person name or an organizational name. for example, a model learned from a dataset in 2017 is difficult to correctly classify articles including "donald trump" or "joe biden" in 2021, because the model does not know a new president change. we call the problem as diachronic bias because it is caused by the difference in the news publishing date in each dataset. this problem occurs with data from even the same domain, making it difficult to construct a robust detection model. this study examines various strategies to mitigate diachronic bias by masking proper nouns that tend to cause the bias, such as names of people and places. first, we analyzed the correlation between labels and phrases in several fake news detection datasets with different creation periods and noted the deviation of words, mainly person names. we then applied and validated several masking methods focusing on proper nouns to mitigate diachronic bias in the tackling of fake news detection tasks. there is a tendency for real labels to be highly correlated with common phrases, while fake labels are highly correlated with person names. ## related work analysis and examination of mitigation methods for various types of bias have been conducted for detecting offensive language and hate speech: author bias @xcite , annotator bias @xcite , gender bias @xcite , racial bias @xcite , political bias @xcite , etc. dayanik and padó (2020) focuses on frequency bias of person name in their dataset, while we handle person name considering the passage of time. in various studies, bias analysis and mitigation are addressed for the fact verification task, where given texts as judged as factual or otherwise from several pieces of evidence, and is one of the recognizing textual entailment tasks. for example, @xcite and @xcite proposed mitigation methods by replacing some words with specific labels to build a robust inference model for out-of-domain data. however, to the best of our knowledge, there has been no study or analysis of bias in fake news detection datasets. ## diachronic bias mitigation 4.1 masking methods we examine multiple masking methods, starting from word deletion to word replacement for input text data to mitigate the diachronic bias and to build a robust detection model for out-of-domain data. we utilize named entity recognition (ner) @xcite in flair @xcite to search for words to be used as masks. examples of each masking method are listed in appendix d. named entity (ne) del: words tagged with nes are removed. this masking method aims to build a detection model independent of ne, same as @xcite . basic ner: words tagged with nes are replaced with the corresponding labels, such as per (person label), loc (location label), etc. wikid: words tagged with per labels are replaced with wikidata (wikidata) label, specifically, position held (p39), or alternatively occupation (p106) depending on availability. for example, the use of wikidata at that time made it possible to replace the phrase "barack obama" in articles from 2015 and "donald trump" in those from 2020 with the same label as president of the united states (q11696). this makes fake news detection models more robust against the passage of time and potentially more effective in mitigating the bias. wikid+del: words tagged with per labels are replaced by the same rule as wikid, and words tagged with other nes are removed. wikid+ner: words tagged with per labels are replaced by the same rule as wikid, and words tagged with other nes are replaced with the corresponding label. ## conclusion this study proposed a new bias concept, diachronic bias, caused by the difference in the creation period of various fake news datasets. we firstly examined the deviation of phrase appearance in respective fake news detection datasets with different creation periods. we then proposed masking methods using wikidata to mitigate the influence of person names. these masking methods achieved higher accuracy in out-of-domain datasets and showed to be made a model more robust. in the future, more sophisticated approaches such as utilizing a knowledge graph to mitigate diachronic bias would be considered for fake news detection models. in addition to diachronic bias, political bias and racial bias are likely to exist in fake news detection datasets; clarifying these biases in detail is an important next research direction.