Connecting the Dots Between Fact Verification and Fake News Detection (2020)

Abstract: Fact verification models have enjoyed a fast advancement in the last two years with the development of pre-trained language models like BERT and the release of large scale datasets such as FEVER. However, the challenging problem of fake news detection has not benefited from the improvement of fact verification models, which is closely related to fake news detection. In this paper, we propose a simple yet effective approach to connect the dots between fact verification and fake news detection. Our approach first employs a text summarization model pre-trained on news corpora to summarize the long news article into a short claim. Then we use a fact verification model pre-trained on the FEVER dataset to detect whether the input news article is real or fake. Our approach makes use of the recent success of fact verification models and enables zero-shot fake news detection, alleviating the need of large scale training data to train fake news detection models. Experimental results on FakenewsNet, a benchmark dataset for fake news detection, demonstrate the effectiveness of our proposed approach.

## introduction recently, fake news has been appearing in large numbers, which are easily accessed and disseminated in the online world with the booming development of online social networks. users can be affected due to the deceptive words intentionally and verifiably false, which makes fake news detection urgent and important for maintaining social order @xcite . most existing methods for detecting fake news rely heavily on supervised learning on a large scale dataset with news articles labeled as fake or real by human experts. however, such a labeled dataset is difficult and time-consuming to obtain while few large scale fake news detection datasets are publicly available @xcite . meanwhile, with the fast development of pre-trained language models @xcite @xcite and the release of large scale datasets @xcite , research on the fact verification task have enjoyed fast advancement @xcite @xcite . however, the advancement in the field of fact verification fails to transfer to the task of fake news detection due to the different nature of input sequences (i.e., fact verification aims to check the reliability of a claim of one or a few sentences while fake news detection aims to check the trustworthy of a long article) and the lack of large scale training data to train the fact verification models for fake news detection. to address the above issues, in this paper, we propose a simple yet effective approach to connect the dots between fact verification and fake news detection. our approach exploits off-the-shelf models in two relatively well-studied problems-text classification and fact verification-to tackle the problem of fake news detection. specifically, our approach first employs a text summarization model pre-trained on news corpora to summarize the long news article into a short claim. then we use a fact verification model pre-trained on the fever dataset to detect whether the input news article is real or fake. our approach transfers the recent success of fact verification models to enable zero-shot fake news detection, alleviating the requirement of large scale datasets to train fake news detection models. experimental results on fakenewsnet, a benchmark dataset for fake news detection, show that our approach can achieve comparable performance with several competitive supervised content-based fake news detection models in a zero-shot fashion. in addition, when fine-tuning the fact verification model with labeled examples, our approach significantly outperforms the supervised baselines. this demonstrates the effectiveness of our proposed approach. ## related work the problem of fake news detection has gained much research interests with the widespread of fake news on social media @xcite . existing fake news detection models can be categorized into content-based models @xcite and social context based models @xcite @xcite . in this paper, we focus on content-based models that enable early fake news detection before the spread of fake news. recent content-based fake news detection models @xcite @xcite generally formulates the problem as a text classification problem. however, many news articles are very long, which hinders the application of state-of-the-art pre-trained language models such as bert @xcite because their maximum context length is generally 512. moreover, few large scale fake news detection datasets are available and they are generally limited in their domains while human annotation of fake news is time-consuming and expensive, which hinders the application of current supervised content-based fake news detection models. another recent work related to our paper is an unsupervised fake news detection method proposed by @xcite . however, their approach is based on the social context, which requires additional information and thus less general to content-based approaches. our approach, in contrast, is the first fake news detection method that enables zero-shot fake news detection without any labeled examples or additional information to the best of our knowledge. ## methodology our approach is a novel content-based fake news detection method that exploits off-the-shelf pre-trained models in well-established text summarization and fact verification problems to enable zero-shot fake news detection without training on labeled datasets. as illustrated in figure 1 , in the first stage, our approach employs a text summarization model pre-trained on large scale news corpus to summarize the input long news article into a short claim which consists of one or a few sentences. afterward, we use a fact verification model pre-trained on fever @xcite , a large scale fact verification dataset, to check the trustworthiness of the article. we then describe the details of the two stages in our proposed approach. for the first stage. we employ an open-sourced bert-based extractive summarization model foot_0 @xcite to summarize the input news article into a short claim. the length of the output summary is controlled to mimic that of the claims in the fever dataset by setting the compression ratio set to 0.1 and selecting the top 2 predicted sentence, which minimizes the inconsistency between the input of the fact verification model used in the second stage during its training and inference procedure. in addition, opinions in long articles are generally scattered, which makes it difficult for a fake news detection model to judge the reliability of long articles. our approach makes the information more concentrated in a few sentences, making it easier to identify the trustworthiness of the article. during the second stage, we employ gear @xcite , a competitive fact verification model based on bert and graph neural network trained on the fever dataset to verify the claim generated during the first stage. however, the model requires multiple support evidence, which is available in the fever dataset, to make the prediction. to mitigate this issue, we propose to construct evidence of the claim using a commercial search engine such as google. concretely, we first extract the keywords in the claim with allennlp foot_1 @xcite , an open-sourced nlp toolkit. afterward, we feed the keywords into a search engine to crawl web texts related to the claim. we filter the urls that contains only pdf or images or the text content is less than 100 tokens, and select support evidence for the claim according to the sentence embedding similarity under sentencebert @xcite , a pre-trained sentence embedding model. we select the top 5 related sentences as evidences and feed them into the pre-trained gear model together with the original claim to predict whether the claim is trustworthy or not. note that the fact verification model pre-trained on fever is a three-way classification model including the "not enough infomation" class. we simply omit this class and compare the output probability of the real and fake class to make the binary classification. in addition, when labeled examples are available, we can further fine-tune the fact verification model pre-trained on the fever dataset in a continual learning fashion with our target dataset to achieve better performance. for the experiments with fine-tuning the fact verification model with labeled fake news detection examples, we initialize the output layer to make the model perform binary classification and use the default hyperparameter of gear to perform the fine-tuning procedure. dataset we adopt fakenewsnet @xcite , a recently released benchmark dataset for fake news detection in our experiments. the dataset consists of news articles in two domains including politifact and gossipcop. we run the official code of the paper to obtain the datasets, detailed statistics of the dataset are presented in ## discussion & conclusion in this paper, we propose a novel fake news detection approach that exploits well-trained text summarization model and fact verification model. by connecting the dots between fact verification and fake news detection, our approach enables zero-shot fake news detection. there exists previous work that exploit models trained on one task for another related task, such as using machine translation models for paraphrasing @xcite and grammatical error correction @xcite . our approach is similar to this line of research but combines two models in other tasks for another downstream task. experiments on a fake news detection benchmark show that our approach can yield comparable performance with competitive supervised content-based fake news detection models in a zero-shot fashion. our approach can also leverage labeled examples more effectively than conventional supervised methods with continual learning, enabling building fake news detection models more effectively for domains where few labeled data are available. one limitation of our approach is that it is based on pre-trained language models and thus can be computationally expensive to use in real-world applications. therefore, for future work, we plan to exploit methods to accelerate the inference of pre-trained language models including model compression methods like distilbert @xcite and bert-of-theseus @xcite , as well as adaptive inference methods such as pabee @xcite .