Automatic Fake News Detection in Political Platforms - A Transformer-based Approach (2021)

Abstract: The dynamics and influence of fake news on Twitter during the 2020 US presidential election remains to be clarified. Here, we use a dataset related to 2020 U.S Election that consists of news articles and tweets on those articles. Therefore, it is extremely important to stop the spread of fake news before it reaches a mass level, which is a big challenge. We propose a novel fake news detection framework that can address this challenge. Our proposed framework exploits the information from news articles and social contexts to detect fake news. The proposed model is based on a Transformer architecture, which can learn useful representations from fake news data and predicts the probability of a news as being fake or real. Experimental results on real-world data show that our model can detect fake news with higher accuracy and much earlier, compared to the baselines.

## introduction fake news refers to false or misleading information that appears as real news @xcite . fake news can be broadly categorized as either misinformation (unintentional false information) or disinformation (deliberate false information). recent social and political events, such as 2020 united states presidential election, have seen an increase in fake news (e. @xcite . according to a report by first draft news 1 , america's current disinformation crisis is the result of more than two decades of corruption in country's information ecosystem. there are many 1 https://firstdraftnews.org/latest/fake-news-complicated/ factors to blame for this social and political misinformation. for example, the role of social media that is unregulated, lack of investment in public media, downfall of local news outlets, and emergence of hyper-partisan online outlets. an information (news) ecosystem consists of publishers (news media that publish the news article), information (news content) and users @xcite . initially, the news comes from the publishers. then, it goes to the news websites, from where it goes to the users who share news on different platforms @xcite . if the news is fake, some users may find it more sensational and interesting to comment on and share over their networks. the existence of the bots in social media makes it even worse, who spread misinformation through multiple channels to urge people believe the fake news. therefore, it becomes crucial to stop the fake news before it reaches to a broad audience. in this paper, we aim to effectively detect the fake news. generally, the content of fake news is vague and misleading (c. @xcite . according to a research @xcite , the content of fake news consists of certain patterns, such as excessive use of capital letters, punctuations, or emotionbearing words, which gives us clues about a news being fake or real. however, if the content of news is not sufficient, then the social contexts may be useful to assess the veracity (truthfulness) of news. the social contexts @xcite refers to users' interactions, such as, comments, shares, likes, followers-followees relations etc., that are helpful to determine if a news fake or real. sometimes, even the verified accounts in social media are involved in the propagation of fake news ## methodology problem definition: given news and associated side information (news-related, social contexts and temporal information), the task is to determine if a news item is fake or real. we consider the fake news detection task as a binary classification problem (news as fake or real). we also consider a multiclass classification (news as fake, real or mixed) in the experiments. proposed model: in our work, we modify the structure of pre-trained bidirectional encoder representations from transformers (bert) @xcite to add side information (in addition to main information). the same methodology can also be applied on other transformers (roberta, xlnet, bart, t5 etc.). we represent each news item n by its title (main information) and side information. (temporal, news-related, and social contexts). we believe that having more information is always beneficial. for instance, the author and source provide us with partisan information (political party as belonging to right or left wing). the temporal information is useful for determining whether a fake news is already spread or just released. similarly, social contexts (tweets) give us additional information about users' reactions on news. we present a novel transformer-based model, faker, as shown in figure 1 . the input to model is news items and associated side information. each news item n has a sequence of words, i.e., 𝑁 = {𝑛 1 , 𝑛 2 , … , 𝑛 𝑙 } where l is length. for each news, we have the accompanying side information, i.e., 𝑆𝐼 = {𝑠𝑖 1 , 𝑠𝑖 2 , … , 𝑠𝑖 𝑙 } . in our work, we consider different types (lexical and non-lexical) of side information, whereas our main information is textual. we use 'word' as a general term to represent any word from n or feature from si. the first layer in faker is the embedding layer. the input to the embedding layer is the sequence of words from each input n or si. the [cls] token is added at the start of the sequence and is later used for the class label prediction. we utilize the token and segment embedding from the bert model to represent the syntax and semantics of each word. similar to (q. chen et al., 2019), we also assume that temporal order exists in sequences. so, we use position encoding @xcite to capture the chronological information in the sequences. in our case, the position value of each word is decided by the timestamp of news publication. the output from the embedding layer is then fed into the next twelve layers in the first encoder block. after the encoding process, we get the output vector for each word from news. the contextualized representation after the first encoder block is 𝑁 ̃= {𝑛 ̃1, 𝑛 ̃2 … , 𝑛 ̃𝑙} for the news and 𝑆𝐼 ̃= {𝑠𝑖 ̃1, 𝑠𝑖 ̃2, … , 𝑠𝑖 ̃𝑙} for the side information ( 𝑠𝑖 ̃ comes from 𝑠𝑖 , the dot above i under 𝑠𝑖 ̃ is hidden under the tilde ~). each word vector from 𝑁 ̃ and 𝑆𝐼 ̃ is then passed to a fusion block. fusion block: inside the fusion block, we represent each piece of information (lexical or nonlexical) from 𝑁 ̃ and 𝑆𝐼 ̃ with a token (word). the x is a textual word, nu is numeric word (feature) and c is a categorical word. inspired by the gating mechanism introduced in @xcite , we first take each feature from the non-lexical data (nu and c) and combine them using a gating mechanism to produce a new nonlexical vector h, as shown in equation ( 1 ): ℎ = 𝑔 𝑐 ⊙ (𝑊 𝑐 𝑐) + 𝑔 𝑛𝑢 ⊙ (𝑊 𝑛𝑢 𝑛𝑢) + 𝑏 ℎ (1) where c is categorical feature, nu is numerical feature, w denotes a weight matrix, 𝑏 denotes a scalar bias, and 𝑔 𝑐 and 𝑔 𝑛𝑢 are the gating vector for c and nu respectively. we may refer to 𝑔 𝑖 as a gating vector for a non-lexical feature i. the 𝑔 𝑖 is fused with x using an activation function r. then it goes into h. the 𝑔 𝑖 is defined in equation ( 2 ): the new sequence 𝑁 ̅ = {𝑛 ̅ 𝐶𝐿𝑆 , 𝑛 ̅ 1 , 𝑛 ̅ 2 , … , 𝑛 ̅ 𝑙 } is then fed as input to the next encoder block. we apply the encoder layers of our model on this sequence 𝑛 ̅ . at the end of the second encoder block, we get the sequence 𝑛 ̿ = {𝑛 ̿ 𝐶𝐿𝑆 , 𝑛 ̿ 1 , 𝑛 ̿ 2 , … , 𝑛 ̿ 𝑙 }. the first element in 𝑛 ̿ is the @xcite token that has the necessary information to predict the class {real, fake} label. therefore, the 𝑛 ̿ 𝐶𝐿𝑆 goes through a final transformation to produce a value which can be used to predict a class label. the dataset also includes over 400,000 embedded tweets found in news articles, which we also employ in our research. we use article ids to create sequences based on available features (in the dataset only provides us the ground truth labels (0-reliable, 1-mixed, 2-unreliable) at source-level. these source-level labels are obtained from mbfc, which considers the dimensions of veracity based on a factuality (credibility) and on conspiracy sources. we use the distant supervision @xcite to assign a label to each news story. in that, first we take the distant (weak) labels provided to each news source and use a weighted scheme to label each news article. the intuition of distant labeling is that the training labels at source-level may be imprecise and partial but can be used to create a strong predictive model. this approach is also suggested in the nela-gt-18 paper @xcite and has shown promising results in a recent work @xcite . after doing the labeling, we get around 37k labels as 'fake', 12.5k labels as 'real' and 32k labels as 'mixed'. to handle the data imbalance problem in in the dataset, we use the undersampling technique @xcite , in which the majority class is made closer to the minority class, by removing records from the majority class. initially, we tried the smote technique, in which the distribution of minority class is increased by replicating some records, but due to limited memory, we opt for under-sampling. evaluation metrics: to assess model perform, we use accuracy acc, precision prec, recall rec and f1-score f1, and area under curve auc. compared to acc, auc is usually better at ranking predictions because auc evaluates model performance across all possible thresholds. we treat the fake news detection as a binary classification problem using labels {'real', 'fake'}, and as multiclass classification using labels {'real', fake' and 'mixed'}. comparison methods: for the baselines, we use: fake-news detection methods • trifn @xcite : a matrix factorization methods that exploits user, news and publisher relationships for fake news detection. in the final settings, we choose the following hyperparameters: the news stories and tweets are on average 500 words, so we choose a sequence length of 500 token. we use padding if the length is shorter and truncation if it is greater. the dimensionality is set to be 768. larger batch sizes did not work at our end due to memory limitation. so, we choose the batch size to be 8. the dropout rate is set be 0.25, epochs 10, learning rate 1e-3 and adam optimizer is chosen for optimization. ## experimental results and analysis we present the results of binary and multiclass classification using acc, f1-score (harmonic mean precision and recall) and auc in ## related work following the 2016 election, google, twitter, and facebook all took steps to combat fake news. facebook and twitter also allow users to mark news stories as fake. a marked news story usually then goes through a manual fact-checking process. manual fact-checking is inefficient for detecting fake news early because it is a time-consuming process, and it is also not scalable to handle a large volume of fake news online. in this paper, we look at automated methods for detecting fake news. the automatic fake news detection methods can be broadly categorized as: content-based and social contexts-based methods. most of the existing content-based detection methods @xcite @xcite use style-based features (e.g., sentence segmentation, tokenization, bag-of-words, latent topics, and pos tagging) or linguistic features (e.g., frequencies of words, case schemes, context-free grammar and syntax etc.,) from news articles to detect fake news. one challenge of content-based techniques is that fake news style, platform, and topics are changing constantly. models trained on one dataset may perform poorly on a new dataset with a different content, style, or language. furthermore, because the target variables in fake news change over time, certain labels become obsolete, while others must be re-labeled. these algorithms also necessitate a massive amount of training data to to solve the issues of content-based methods, the researchers begin focusing on social contexts to detect fake news. the existing social contextsbased approaches are categorized into two types: (i) stance-based methods, and (ii) propagationbased methods. the stance-based approaches exploit the users' viewpoints from social media posts to determine the truth @xcite @xcite . the propagation-based methods @xcite @xcite utilize the information related to the dissemination of fake news, e.g., how users spread it. these methods use techniques such as graphs and multi-dimensional points for fake news detection @xcite . while social context methods are useful when there is a lack of news content, they also introduce additional challenges. gathering social contexts, for example, is a broad topic. the data for social contexts is not only large, but also incomplete, noisy, and unstructured, which may render existing detection algorithms ineffective. fake news detection is a subtask of text classification (c. @xcite , which is solved by various machine learning and deep learning methods. some work (y. @xcite uses rnn and cnn networks to build propagation paths for detecting the fake news. some other work @xcite uses matrix factorization methods to detect fake news. a few works @xcite use lstm networks on users' comments to explain if a news is real or fake. a few works @xcite also uses graph networks to propose an explainable fake news detection system. in recent years, there has been a greater focus in nlp research using the transformer models, such as bert @xcite . bert is used in some fake news detection models @xcite @xcite to classify the news as real or fake. despite the robust design proposed in these models, a few limitations are noted. first, these models do not consider a richer set of features from the news items and social contexts. second, the focus in these methods is not on early fake news detection. the inclusion of temporal information is important to early detect fake news (y. @xcite . also the inclusion of side (meta-data) information related to news or social contexts is important to understand the nature of fake news data @xcite . recently, an exploratory study @xcite on fake news gives us more new insights about the timeline of misinformation. in our work, we consider both the temporal and side information to detect fake news. the existing works on fake news focus either on news content or social contexts to detect fake news, we consider both in our work. compared to some previous works @xcite @xcite ) that consider both these aspects, we include a wider set of news-related as well as social context (tweets). a few works (y. @xcite propose early detection of fake news. compared to these methods, we can detect fake detect the fake news much earlier (i.e., after a few minutes of news propagation). compared to the previous works, we consider the latest state-of-the-art neural architectures (transformers). ## conclusion and recommendations in our work, we propose a transformer-based architecture for fake news detection. we utilize the news content and social contexts to detect the patterns of fake news. we also early detect the fake news through a position-aware encoding. we achieve higher performance compared to the baselines, which shows the usefulness of our proposed approach. in addition to fake news detection, this model can also serve for general classification tasks. to further improve the proposed method, a recommendation is to consider more social contexts, such as friends' networks, propagation paths and implicit users' feedbacks. it would also be very useful to consider malicious social media users' profiles and their activities. another recommendation is to combat data and concept drifts. it would also be very useful to understand the tactics of fake news producers in real-time scenarios. furthermore, data labelling scheme can be investigated because of the possibility of incorrectly labelled data, which may lead to data biases (kishore shahi, 2020). a possible extension of this work is to mitigate those biases. we also want to break filter bubbles and burst echo chambers created due to the spread of fake news.