Fake or Real? A Study of A rabic Satirical Fake News (2020)

Abstract: One very common type of fake news is satire which comes in a form of a news website or an online platform that parodies reputable real news agencies to create a sarcastic version of reality. This type of fake news is often disseminated by individuals on their online platforms as it has a much stronger effect in delivering criticism than through a straightforward message. However, when the satirical text is disseminated via social media without mention of its source, it can be mistaken for real news. This study conducts several exploratory analyses to identify the linguistic properties of Arabic fake news with satirical content. It shows that although it parodies real news, Arabic satirical news has distinguishing features on the lexico-grammatical level. We exploit these features to build a number of machine learning models capable of identifying satirical fake news with an accuracy of up to 98.6%. The study introduces a new dataset (3185 articles) scraped from two Arabic satirical news websites (‘Al-Hudood’ and ‘Al-Ahram Al-Mexici’) which consists of fake news. The real news dataset consists of 3710 articles collected from three official news sites: the ‘BBC-Arabic’, the ‘CNN-Arabic’ and ‘Al-Jazeera news’. Both datasets are concerned with political issues related to the Middle East.

## introduction in september 2015, the egyptian army apache helicopters blasted a convoy of mexican tourists in the sinai peninsula which resulted in the killing of 12 tourists as it mistakenly took their four-wheel vehicles for terrorist targets foot_0 . days later, two renowned egyptian talk-show hosts announced that the mexican president stated that he completely understands the egyptian authorities' intention behind the killing of the mexican tourists as egypt has the right to defend its land against possible terrorist threats. a hashtag "#thankyou henrico iglassios" went viral on twitter by egyptian pro-government twitter accounts in an attempt to show gratitude to the mexican president for his understanding. the absurdity of the mexican president's statement was realised a little too late. it was soon discovered that the mexican president is not actually called 'henrico iglassios' and that this statement was never issued by any of the mexican authorities who were in reality extremely angry at the incident. the source of all this commotion was an article published on an online news website, 'al-ahram al-mexici' 2 . the latter is a satirical fake newspaper that constantly posts a mockery of current events in a style that meticulously imitates the style and content of egyptian government news agencies. the egyptian-mexican incident is not the first in egypt or in other parts of the world. there are similar incidents where satirical news is mistakenly propagated by officials or social media users because they either missed the satirical gesture or they did not investigate the source. for instance, the daily beast 3 compiled a list of nine times the headlines of the satirical online newspaper, the onion 4 , fooled famous news outlets into believing their satirical stories. among those duped were the new york times, the us capitol police, and the washington post @xcite . although satirical news is essentially entertaining, it is potentially deceptive and harmful since the satirical cues could be easily missed specially if the news source is obliterated in a shared social media post or tweet. moreover, satirical news aims to entertain by crafting unusual or surprising report of current events. according to a ny times study on the psychology of content sharing, the motivation of almost 50% of online news sharers is to bring "entertaining content to others" @xcite . this makes the deceptive content of a satirical article more likely to be disseminated online especially when the language is a perfect parody of real news and when the original source is not mentioned. in this research, we employ inferential statistics to investigate the defining features of arabic satirical news on the lexico-grammatical level. we also put forward the hypothesis that arabic satirical fake news can be automatically identified with very high accuracy. we demonstrate the effectiveness of using pretrained word embeddings as input to statistical and neural models in detecting arabic fake news articles with a satirical intent. to achieve its objectives, the study presents in section 2 a brief literature review of satirical fake news detection in general and arabic satire detection in particular. section 3 explains the steps followed for the compilation of the corpora used in this study and the preprocessing applied to the datasets. section 4 presents a linguistic-statistical analysis of features specific to arabic satirical fake news. section 5 describes the classification experiments and their evaluation. section 6 presents an error analysis and a discussion of the classification features. section 7 presents a conclusion on the conducted experiments and the linguistic analyses carried out in the study. an early attempt to automatically detect satirical fake news has been conducted by @xcite . their contribution lies in constructing a comprehensive set of linguistic features which they present as satire-defining. although they achieve a high classification accuracy using the standard bag-of-words approach (94.3%), they improve their accuracy by integrating three major feature sets that they discovered to be peculiar of satirical fake news. these feature sets are: 1) profanity, 2) slang, and 3) semantic invalidity. the semantic invalidity of an article is measured by how likely a named entity is mentioned in an unusual context, which is a common feature of satire @xcite . in a more recent study, @xcite proposes a svm-based algorithm enriched with satire predicative features to classify satirical and legitimate news articles. they have shown that absurdity in an english satirical news is achieved by a number of linguistic features on the word-level such as use of slang, co-occurrence of named-entities that are not usually related and the presence of unfamiliar named-entities at the end of news articles as an ironic non sequitur @xcite . the largest data for satirical fake news detection is developed by @xcite who utilise more than 16,000 satirical news articles collected from 14 satirical websites such as the onion and the beaverton 5 . the true news dataset consists of more than 160,000 articles from google news. they develop a neural network model that analyses a hierarchy of character-word-paragraph-document. to decide which paragraph has a satirical cue, they calculate the satirical degree of a paragraph by comparing it against a learnable satire-aware vector which is based on a psycho-linguistic and stylistic feature set. their model suggests that english satirical fake news can be predicted automatically with fairly high precision and an accuracy of up to 98.54% @xcite . a further development on the dataset introduced by @xcite is provided by de sarkar et al. @xcite . they put forward the claim that pretrained word embeddings alone as input are sufficient in detecting satirical articles and that word-level syntax information only marginally improves classifiers performance. they point out that on the lexical level, the english satirical news article is distinct because of its use of profanity, slang, negative affect, atypical grammar structures and punctuation. they acknowledge that hand-crafted analysis of these features might contribute to a robust classifier, but a neural network model such as a convolutional neural network (cnn) or a recurrent neural network (rnn) trained on word embeddings is sufficient for detecting satirical fake news. their analysis of the learned models reveals that news satire is decided by a key sentence which usually comes last. their best classification accuracy is 98.18% achieved by an lstm model using english glove embeddings @xcite . rashkin et al. (2017) also uses a large corpus of 14,170 satirical articles from one source, the onion and add a small number of satirical articles (800) from two other sources. their fake dataset also includes hoax and propaganda articles collected from different sources. their model is based on hand-crafted analyses of linguistic features on the word-level. they deploy the psycholinguistic lexicon provided by the linguistic inquiry and word count software (liwc) to categorise words into groups according to their social function @xcite . they noticed that one distinctive feature of english satirical news is its prominent use of adverbs whereas true news uses assertive words and is less likely to use hedging words. their model uses a max-entropy classifier achieving an f1 score of 0.65. they achieve a relatively lower accuracy by comparison to other satirical news detection research because their fake news dataset includes both hoax and propaganda @xcite . ## data compiling and preprocessing the fake news dataset foot_3 used in the present study is a collection of 3185 crawled articles from two different sites which explicitly declare that they publish satirical fake news. 7 the first site is 'al-hudood' foot_5 (the limits) which is a satirical online newspaper launched in 2013 by a 34-year-old jordanian-palestinian. it is characterised by deadpan headlines reminiscent of the us publication the onion. the site averages 1 million unique visitors a month ("al-hudood", 2019). the second source, 'al-ahram al-mexici' foot_6 (the mexican ahram) website which is a satirical parody of the main national egyptian newspaper 'al-ahram' (alahram-almexici, 2020). the two satirical online newspapers mock political leaders and middle eastern political events. the real news dataset comprises 3710 articles from the bbc-news arabic and cnn-arabic news datasets which are available on the sourceforge part of the arabic computational linguistics project foot_7 as well as the al-jazeera news open-source dataset used in el @xcite . to ensure that the fake and real datasets talk about similar themes, we chose the articles that belong to the politics section in the three real news corpora. a number of preprocessing operations were conducted on the datasets. by surveying the corpora, it was observed that there are phrases typically repeated at the header and footer of each article and in some instances at the middle. for example, "bbc arabic", " " (middle east), " " (give your opinion) are typical of the bbc news articles. the fake news dataset header and footer repeatedly mentioned phrases such as " " (limits), " " (above mentioned), " " (special for the limits), and so on. in order to capture these stylistic guidelines, each dataset was merged into one file and frequency dictionaries of unigrams, bigrams and trigrams were created for each file. the top most 10% of the frequency dictionaries of each dataset included most of the website-specific phrases. two lists of website-specific phrases indicative of the document source were manually created for each dataset and items in the lists were replaced by a space in the corpus. the cleaning of the datasets also included deletion of formatting quirks specific of each source, non-informative textual features such as special characters, english characters and diacritics. moreover, arabic is a quite morphologically rich language where a stem word can be packed with affixes that realise different grammatical functions. hence segmentation of words into their constituent tokens is a very common practice in arabic nlp research. accordingly, the arabicsos, an arabic language segmenter was employed to segment both datasets @xcite . the classification task was performed on segmented and non-segmented versions of the two datasets. ## linguistic-statistical analysis after cleaning the data, a preliminary review of the corpus was conducted to detect category-defining features of the fake dataset. then inferential statistic tests were performed to decide the degree of significance each of these features may have on the classification task. ## classification models we used a supervised classification approach for the fake news detection task. we developed three types of models: baseline models which rely on statistical machine learning models (section 5.1), classic machine learning models that achieved promising performance in other text classification tasks (section 5.2) and neural models that achieved state-of-the-art results in satirical news detection (section 5.3). we evaluated the models' performance on a held-out test set by measuring the macro average of the precision, recall and f1 score of the two categories (real and fake) as well as the total percent accuracy of each model. ## error evaluation and feature analysis this section presents an error analysis of the false negative instances where satirical news was mistaken for real. they constituted the higher number of classification errors with all the models. we also explore the most informative features for the classification task. both analyses provided useful insights into the problem of detecting arabic satirical fake news. ## conclusion this paper proposed a novel classification problem for arabic computational linguistics and machine learning: detecting whether an arabic news article is true or satirical. the study experimented with different classification techniques including state-of-the-art deep learning models. experimental results showed a superior performance of a proposed architecture employing the combination of a cnn with pretrained word embeddings. a linguistic-statistical analysis of the satirical dataset gave a deeper insight into the phenomenon of arabic satirical news. however, the high accuracy of the cnn neural architecture suggests that pre-trained word embeddings as input, without the aid of any syntactic information or any hand-crafted linguistic features, suffices in capturing arabic satire. the reason is that a satirical cue is always there in a fake satirical article. although the writer's aim is for the absurdity of the news to be ultimately realised, the satirical message can still be missed and taken for real. it is worth mentioning, that the classifier successfully detected the deceptive content of the satirical fake news article published by al-ahram al-mexici which was the cause of the diplomatic crisis between egypt and mexico mentioned in the introduction of this paper. the study gives promising evidence that the dissemination of this type of deceptive content via the internet can indeed be automatically prevented by ai solutions.