Automatic Detection of H ungarian Clickbait and Entertaining Fake News (2020)

Abstract: Online news do not always come from reliable sources and they are not always even realistic. The constantly growing number of online textual data has raised the need for detecting deception and bias in texts from different domains recently. In this paper, we identify different types of unrealistic news (clickbait and fake news written for entertainment purposes) written in Hungarian on the basis of a rich feature set and with the help of machine learning methods. Our tool achieves competitive scores: it is able to classify clickbait, fake news written for entertainment purposes and real news with an accuracy of over 80%. It is also highlighted that morphological features perform the best in this classification task.

## introduction the growing number of online news and the ability to easily and rapidly distribute information on the internet increasingly stimulate demand for automatic fact checking @xcite . as a consequence, linguistic aspects of deception, bias and uncertainty detection have raised worldwide interest recently and have been thoroughly studied in a variety of nlp-applications @xcite @xcite @xcite . however, determining the trustworthiness of news and separating facts from misinformation is still a challenging and often controversial task @xcite . there may be several motivations for spreading fake news on the internet. hoax websites, for instance, publish dubious news (clickbait) in order to spread different kinds of misinformation or make money by spreading commercials. to be more specific, fabricated news draw disproportionate attention on social networks most of the time, outperforming conventional news @xcite . consequently, publishing interesting fake news is a great way to spread advertisements on the internet. at the same time, there are pages where the primary purpose is to entertain readers by spreading fake news. in this case the readers are aware of the deceptive nature of the information provided and they read it just for fun. in this paper we will focus on two types of fake news written in hungarian, namely clickbait and fake news written for entertainment purposes. our main goal is to distinguish these two types from real news with machine learning methods. we will also analyse what type of linguistic information can contribute the most to performance. the paper is structured as follows: first, we will give a short review of the relevant research work and we detail the importance and benefits of the recent analysis. then, we will present the corpus analysed, along with its basic statistical data and the methods and tools we used in the experiments. next, we will introduce our rich feature set consisting of statistical, morphological, syntactic, semantic and pragmatic features applied for statistical significance analysis and machine learning experiments. we will discuss the findings of the significance analysis, and then, we will provide a detailed description of the machine learning experiments, along with the results. finally, we discuss the results and add some ideas for future work. ## related work most of the authors address the issue of automatic fact extraction and binary classification verification task based on machine learning methods and annotated datasets. there are several studies addressing the phenomena of deception and bias in different types of discourse. for instance, most of the authors analyze the phenomena of deception and bias either in speech text @xcite @xcite or address the issue of automatic fact extraction and binary classification verification task @xcite @xcite @xcite . in addition, uncertainty detectors have mostly been developed in the biological and medical domains @xcite . also, a few studies seem to address the issue of the systematic analysis of a huge amount of propaganda texts @xcite @xcite @xcite . recently, several databases have been compiled for the english language which contain both fake and real news. for instance, the dataset described in vlachos and riedel (2014) consists of a total of 221 statements, along with their veracity value. the liar corpus contains approximately 13,000 short political statements @xcite . the emergent corpus consists of 300 statements and 2500 news related to the semantic content of the statements, along with their veracity value @xcite . our study is most similar in vein to @xcite , which compares satirical news to real news, collected from different websites. @xcite used crowdsourcing to generate fake news on the basis of real news and then carried out machine learning experiments to separate the two types. there are also a few studies that identify clickbaits (see e.g. @xcite for english and @xcite for bulgarian). the above studies focus on the phenomena of deception and bias almost exclusively in sources written in english, so the reported research findings and models are mostly limited to the english language. the same goes for the currently existing datasets. @xcite forms an exception, which, however, aims at the distinction of brazilian portuguese fake and real news. to the best of our knowledge, our recent research work is the first attempt at the automatic detection of hungarian fake news. the main contributions of our paper are the following: ## the corpus our study was conducted on a corpus of 180 online news compiled by us. first, the real news come from several national and regional news portals, e.g. www.index.hu , www.origo.hu and www. delmagyar.hu . second, the fake news were collected from two sources. on the one hand, we selected some special news published on the 1st of april (fools' day) on some of the news portals basically spreading real news, as this day is traditionally seen as an occasion for making pranks. these news were intended to play a trick on readers. on the other hand, we downloaded articles from the hírcsárda website foot_1 . these texts are mostly based on parodies of current political and public events, and their explicit purpose is not to spread real or fake news but to entertain the readers. third, the clickbait news were downloaded from websites that were collected by a hungarian real news portal and were claimed to be unreliable foot_2 . most of the texts were published during the time period between 2017 and 2019. for each type of text, we collected 60 documents. the news were randomly selected but we made sure that a given topic should not be included in the corpus more than one time. the basic data of our corpus are presented in ## experiments in this section, we present our methods for identifying the three types of news with machine learning methods as well as providing statistical significance analysis for the linguistic features defined for the task. ## results in this section, we present the results of both our statistical analysis and machine learning experiments. ## discussion of the results our results showed that our machine learning experiments achieved an accuracy of 82.78% in the classification task of real, clickbait and entertaining fake news. in this case we used all the feature groups with the exception of syntactic features. here we discuss our results, with special regard to morphological and semantic features. moreover, we also provide an error analysis. ## conclusions and future work in this work, we reported on the automatic discrimination between hungarian real news, clickbait news and fake news written for entertaining purposes. our results confirm that it is possible to successfully detect untrustful news based on a rich -morphological, syntactic, semantic and pragmatic -feature set, and especially morphological features play an important role in the process. besides morphological features, the added value of semantic features was also apparent, while at the same time, syntactic features did not have a notable effect on our results. our experiments show the potential for exploiting the morphological, more specifically part-of-speech features for fake news detection. as a next step of the research, on the basis of the findings of the recent analysis, we would like to apply our methods and tools on texts belonging to other domains. we will attempt to further train our algorithm to discriminate texts containing propagandistic features, bias, misinformation or disinformation. moreover, we would like to compare these results to previous findings concerning the linguistic features of hungarian communist propaganda texts @xcite . finally, we would like to compare our results obtained on hungarian texts with those on english corpora @xcite and possibly other languages as well.