Fake news detection for the R ussian language (2020)

Abstract: In this paper, we trained and compared different models for fake news detection in Russian. For this task, we used such language features as bag-of-n-grams and bag of Rhetorical Structure Theory features, and BERT embeddings. We also compared the score of our models with the human score on this task and showed that our models deal with fake news detection better. We investigated the nature of fake news by dividing it into two non-overlapping classes: satire and fake news. As a result, we obtained the set of models for fake news detection; the best of these models achieved 0.889 F1-score on the test set for 2 classes and 0.9076 F1-score on 3 classes task.

## introduction fake news detection becomes a more significant task. it is connected with an increasing number of news in media and social media. to prevent rumors and misinformation from spreading in this news, we need to have a system able to detect fake news. it also could be useful because it's hard for people to recognize fake news. approaches to this task are being developed for english. however, fake news texts can be written originally in different 'source' languages, i.e. in russian. to tackle such content in social media, multilingual systems might be used. for russian, only preliminary research for automated fake news detection was undertaken before. moreover, fake news detection in russian becomes more actual due to the appearance of new laws in the russian legal system. for example, since 2020 the russian legal system has special criminal law 1 for spreading fake news about emergencies. furthermore, due to an increasing number of fake news, connected with russia, it will be useful to have the fake news detection module for the russian language to check original news in russian. in our paper, we trained and compared different models for fake news detection in russian. we checked if various language features alone can be helpful for this task. as baseline models, we used support vector machines (svm) and logistic regression over a bag-of-n-grams. also, we trained similar models, but over features obtained from the discourse parsing of news (rhetorical structure theory (rst) discourse features, as in @xcite ). for each news text, these features were constructed from its hierarchical discourse tree representation. it contains discourse @xcite relations between text segments -discourse units, starting with the smallest 'leaf' segments -elementary discourse units. as the third model, we fine-tuned bert (for the russian language) for the fake news detection task. we investigated all existing datasets for automated fake news detection for russian. during our study, we also examined some hypotheses about different types of fake news. we divided our data into three parts -non-satirical fake news (that are equal to simple fake news), satirical fake news, and real news. the satirical fake news is the deceptive news, that was written deliberately with humorous purpose or at least without disinformation purpose. it is presented in the format and style of legitimate news articles @xcite . a more detailed description of the data splitting can be found in section 3. we established that, for the classification task, satirical news should be singled out as a separate class, among fake news and real news. finally, we annotated the part of the dataset manually and compared the results of our models with human performance, setting goals for future research steps. as for discourse features, @xcite incorporate hierarchical discourse-level structures for fake news detection. structure-related properties (number of leaf nodes, preorder difference, parentchild distance) identify structural differences between fake and real news texts. the approach, based on discourse dependency trees, yields better results (82.19%) than approaches based on n-grams, on liwc features, on rst discourse features taking the hierarchical structure of document into account (as in @xcite ), or bigrnn-cnn and lstm approaches using word embeddings in sentences. @xcite propose a set of various features for context and discourse modeling. there are also discourse features among them, based on automated discourse parsing according to rst. they are focused on the direct relationship between a target sentence and other sentences in a segment and on the internal structure of a target sentence (number of nuclei and satellites). such rhetorical relation types as background, enablement, elaboration, attribution are associated with factually-true examples. linguistic features for fake news detection have limitations and can be used in addition to automated fact-checking. i.e., shuster @xcite study linguistic (stylistic) features for machinegenerated misinformation detection and conclude that they are limited in detecting if texts generated by language models are fake, as such texts hide stylistic differences between falsified and truthful content. automated satire and biased text detection are closely connected tasks. for satire detection, absurdity feature (unexpected introduction of new named entities within the final sentence) @xcite , pos features @xcite @xcite , psycholinguistic, readability and structural text features @xcite , sentiment scores and named entity features @xcite can be used. satire can be distinguished from fake news using semantic representation with the bert language model and with linguistic features based on textual coherence metrics that also include basic language features, such as readability features, sentence length, number of words @xcite . language features, without checking external sources of information, are also mainly used for biased language and propaganda detection @xcite 2. claims verification. in automated fact-checking for english, @xcite @xcite @xcite consider evidence detection and claims verification. several recent studies are focused on evidence detection for explainable claim verification, based on semantic entailments for claims @xcite , incorporating semantic similarity between comments and claims @xcite . bert language model @xcite can be used for checking if a comment is factual @xcite , for retrieving evidence sentences and verifying the truthfulness of claims against the retrieved evidence sentences @xcite ; there are also experiments on using it solely, without any external knowledge or explicit retrieval components @xcite . there are several datasets, i.e. pheme @xcite , liar (wang, 2017), rumoureval @xcite , fever @xcite . for russian, few initial research studies on fake news detection have been conducted, each one based on a single dataset. basic lexical, syntactic, and discourse parameters were examined in @xcite . the impact of named entities, verbs, and numbers was investigated in @xcite . there are only a few datasets for fake news language and no existing datasets for claims verification. to build a misinformation detection system for social media posts in russian, firstly we study language features for fake news detection. as discourse features were already regarded for fake news detection for russian, we check their impact in our study too. ## data 3.1 data collection to the best of our knowledge, there are three available datasets for fake news detection in russian. we incorporate them for our research and base it on them. in our dataset, we used only one source of satirical news because we found only one reliable source of satirical fake news. it could induce some bias during model training. on the other hand, news from this source are written by different authors on various topics, so we suppose that one source could be enough. ## error analysis and discussion as an automated fake news detection system could be used as a preliminary filter before human evaluation, it is important to reduce the number of false positives for fakes and satire. for the selected bert model, the false positives rate is only 0.05 for fakes and 0.06 for satire on the test set (0.05 and 0.05 on its human-annotated part). for the n-grams model, the false positives rate is 0.01 for fakes and 0.02 for satire on the test set (0.02 and 0.03 on its human-annotated part). we performed an error analysis on the annotated part of the test set, to compare the model's results with human assessments. metrics on this part slightly differ from the metrics on the whole test set ( we also looked at human labels and n-grams model results on the annotated part of the test set ( ## conclusions the research can be considered as the first step in building an automated state-of-the-art system for russian that could detect fake and satirical news and perform automated fact-checking. in this step, we studied language features for fake news and satire detection. we trained and compared different models for fake news detection in russian, based on all existing available datasets for the russian language. we investigated bag-of-n-grams features, bag of rst features, and bert embeddings. we found out that satirical news should be singled out as a separate class, among fake news and real news. we also compared the score of our models with the human score. the best bert-based model achieved a 82.2% f1-score and 74.8% accuracy score on a 3 class classification task, which is bigger than the mean human result, but less than the metrics for the bag-of-n-grams based model, which achieved 90.8% f-score and 94.1% accuracy. we found that "bag-of-rst" features do not improve the performance of the bag-of-n-grams model, as they have reached almost the same scores on the test set. further feature importance analysis and hyperparameter results analysis showed that unigrams are the most important features for fake news detection on our dataset. the model outperforms human evaluation results based on the majority vote. for automated fake news detection, a combination of different methods should be applied. in future studies, the claims verification module for russian should be developed and used together with the linguistic features models. new social media datasets of fake, satirical, biased, and hyperpartisan news for russian should be collected and annotated according to detailed guidelines. we are also planning to try self-supervision methods for extending the datasets. after that, experiments should be performed on creating models for biased and hyperpartisan content, satirical content detection. wider sets of language and content features should be used for them. we are also going to use multilingual sentence embeddings and transfer learning techniques, in order to incorporate the existing models and approaches, developed for automated claims verification for english, to this task for russian.