Data Augmentation for Fake News Detection by Combining Seq2seq and NLI (2023)

Abstract: State-of-the-art data augmentation methods help improve the generalization of deep learning models. However, these methods often generate examples that contradict the preserving class labels. This is crucial for some natural language processing tasks, such as fake news detection. In this work, we combine sequence-to-sequence and natural language inference models for data augmentation in the fake news detection domain using short news texts, such as tweets and news titles. This approach allows us to generate new training examples that do not contradict facts from the original texts. We use the non-entailment probability for the pair of the original and generated texts as a loss function for a transformer-based sequence-to-sequence model. The proposed approach has demonstrated the effectiveness on three classification benchmarks in fake news detection in terms of the F1-score macro and ROC AUC. Moreover, we showed that our approach retains the class label of the original text more accurately than other transformer-based methods.

## introduction the modern world provides great opportunities for news spreading. news travels fast, and it is difficult to expeditiously confirm or deny its credibility. in this regard, there is evidence that the tools for detecting fake news play a crucial role in the regulation of information flows. although machine learning models are widely used in fighting fake news, their performance depends on the size and quality of training data. collection and annotation of text corpora require significant time costs. as an interim solution, augmented data obtained from a small number of annotated texts can be used while training. data augmentation (da) is the artificial creation of training data for machine learning by transformations @xcite . even though the cur-rent state-of-the-art da methods show impressive results, they are still ill-suited for some natural language processing tasks, such as fake news detection. the bottleneck is non-conditional da that contradicts the preserving class labels. thus, the generated news seems to be untruthful. neither rule-based nor model-based approaches guarantee the factual consistency of the original and generated text. this can be a challenge for practical applications because the system will input fakes as examples of real news, and vice versa. in this paper, we propose a da approach that enables the generation of training examples flowing logically from the original texts. to that end, we combine pre-trained sequence-to-sequence (seq2seq) models showing sota results in da, with natural language inference (nli) models estimating textual entailment information. the task of nli is to predict an entailment relation label (output) given a premise-hypothesis pair (input) @xcite . the contribution of this paper is two-fold: a) we built a model to augment data in the field of fake news detection by combining seq2seq and nli. the model allows us to generate coherent outputs for original data; b) we evaluated and compared several approaches to da on three datasets for fake news detection. the paper is organized as follows. section 2 contains a brief review of related work. section 3 describes the proposed approach. in section 4, we provide the details of the experimental setup. we report the results in section 5. section 6 concludes this paper. ## results and discussion we report the results for all classifiers in terms of the f1-score macro (f1) and roc auc (roc). for all corpora, we used five-fold cross-validation to obtain more reliable scores. first, we evaluated the classification performance for the models trained on original corpora. during cross-validation, we consistently split the original corpus into training and test subsets five times. we added generated data to the training subset and shuffled the extended training subset. for each dataset, we generated n texts (n is the training subset size). therefore, the training subset size increased to (2 × n) after da. the model was evaluated on the test subset. as can be seen from the table, in the majority of cases, da methods increase the classification performance. the results of transformer-based methods are mostly higher than the results of eda and bt. the best result for the coaid dataset in terms of f1 was shown using the original corpus. probably, the effect of transformer-based data augmentation for this dataset could be improved using the models pre-trained on medical corpora. although several da models show close results, bart-nli outperforms other methods on fa-kes (f1), coaid (roc), and liar (f1). cbert shows the best scores on fa-kes (roc) and liar (roc). hence, the proposed model outperforms other methods in three of the six cases. in two of the six cases, it demonstrates the second best results (fa-kes, roc and liar, roc). for coaid and f1-score, bart-nli demonstrated only a fifth result out of six, probably because of the absence of domain-adaptive pretraining of roberta-mnli. compared to bart, bart-nli increased the results for all datasets in terms of both the f1-score and roc auc. further, we evaluated the semantic fidelity of the generated texts @xcite . we trained a classifier on each corpus and used the trained classifier to predict the label of the generated output ( the scores obtained by bart-nli are significantly higher than the results of other transformer-based methods. ## conclusion in this paper, we propose an approach to combine seq2seq and nli models to improve the coherence of generated texts in da. the approach showed relatively high results on three datasets for fake news detection. for all considered datasets and both metrics, the proposed approach improved the results of bart for da in fake news detection. we compared the results with several common baselines and demonstrated that our approach preserves the language semantics compared to other transformerbased da methods. in general, our results show the effectiveness of the use of nli models to generate new training data to detect fake news. general limitations of the study include increasing time and memory costs in comparison with using bart separately. moreover, for some datasets, the performance of the approach is much lower than for others. an important feature of the study is the use of short texts. the effectiveness of the approach for longer texts requires further investigation. the proposed approach probably can be applied in other subject areas where augmented data should be coherent with original texts. in the future, we will explore how to perform text da using seq2seq and nli for different natural language processing tasks, various models, and longer texts. in addition, the effect of domain-specific pre-training can be explored.