CUETS entiment S illies@ D ravidian L ang T ech EACL 2024: Transformer-based Approach for Detecting and Categorizing Fake News in M alayalam Language (2024)

Abstract: Fake news misleads people and may lead to real-world miscommunication and injury. Removing misinformation encourages critical thinking, democracy, and the prevention of hatred, fear, and misunderstanding. Identifying and removing fake news and developing a detection system is essential for reliable, accurate, and clear information. Therefore, a shared task was organized to detect fake news in Malayalam. This paper presents a system developed for the shared task of detecting and classifying fake news in Malayalam. The approach involves a combination of machine learning models (LR, DT, RF, MNB), deep learning models (CNN, BiLSTM, CNN+BiLSTM), and transformer-based models (Indic-BERT, XLMR, Malayalam-BERT, m-BERT) for both subtasks. The experimental results demonstrate that transformer-based models, specifically m- BERT and Malayalam-BERT, outperformed others. The m-BERT model achieved superior performance in subtask 1 with macro F1-scores of 0.84, and Malayalam-BERT outperformed the other models in subtask 2 with macro F1- scores of 0.496, securing us the 5th and 2nd positions in subtask 1 and subtask 2, respectively.

## introduction social media has fundamentally transformed how we receive and exchange information in the digital era. but social media is also a source of false news, misinformation, and content emphasized by sensationalism, manipulation, and propaganda @xcite . so the adoption of social media is very significant for awareness, but the authenticity of news is the cause of concern as some sources of news are not reliable @xcite . however, incorrect information may swiftly spread, sway public opinion, cause conflict, and advance agendas. social media fake news undermines truth, democracy, and social unity @xcite . propaganda raises public safety concerns. financial losses and stock market fluctuations can result from rumors or false information about companies. so to maintain social cohesion, protect against cyber threats, and promote ethical journalism, it is important to detect fake news. sometimes fake news can hamper the reputation of individuals, organizations, or businesses. so it is important to identify and correct false information to protect the integrity of affected people. therefore, automated fake news identification is of utmost priority in today's digital age. fake news detection has been a prominent subject of study, with academics examining different methodologies, databases, and nlp solutions to handle this issue @xcite . this work aims to develop a system that can classify news into original and fake for subtask 1 and classify a text into four predefined categories for subtask 2. the key contributions of this work are illustrated in the following: ## related work recent studies have made significant strides in detecting fake news in dravidian languages. a dravidian dataset was introduced by @xcite , and they utilized unique adaptive learning to finetune transformer models. their work demonstrated the effectiveness of transfer learning algorithms, with transformer models, particularly m-bert and xlm-roberta, outperforming other approaches. in another study, transformer models, including m-bert, al-bert, bert, and xlnet, were investigated by @xcite to detect fraudulent content. among these models, m-bert exhibited the best performance. bala and krishnamurthy (2023) employed google's muril model with a curated dataset of labeled dravidian data to detect fake news. by leveraging fine-tuning techniques, their work showcased the effectiveness of the "mural-base-cased" model in identifying fake news. to detect fake news in malayalam, @xcite used lr, mnb, and an ensemble model (mnb, lr, and svm). among the three models, the ensemble model performed the best with a macro f1-score of 0.831. kumari et al. (2023) utilized fine-tuning techniques on the indicbert model (macro f1 score of 0.78) for detecting misinformation in dravidian languages. they employed sbert sentence embedding, dnn-based classification, and an ensemble classifier to accurately categorize text. chakravarthi et al. (2023) focused on categorizing code-mixed social media comments and posts into offensive or not offensive at different levels and presented a multilingual mpnet and cnn fusion model with weighted average f1-scores of 0.85, 0.98, and 0.76 for tamil, malayalam, and kannada, respectively. kaliyar et al. (2021) proposed fakebert, a bert-based deep learning strategy, to identify bogus news. they also employed deep learningbased models, including cnn and lstm. the proposed fakebert model outperformed the other models with an accuracy of 0.989. hossain et al. (2022) employed logistic regression to detect the abusive language in tamil text. the lr and cnn+bilstm models outperformed the others, with lr achieving a higher recall value (0.44) than cnn+bilstm (0.36). for the fake news detection task in the urdu language, @xcite utilized an ensemble of transformer models. @xcite proposed a multilingual ensemble-based model for identifying offensive content in low-resource dravidian languages. the mode achieved an f1-score of 0.97, 0.75, and 0.70 for the malayalam, tamil, and kannada datasets, respectively. monti et al. (2019) proposed a novel automatic fake news detection model based on geometric deep learning. the authors achieved high accuracy for fake news detection with an roc auc score of 92.7%. ## task and dataset description this shared task @xcite was organized by the organizers to detect and classify fake news. the shared task foot_0 included two subtasks: subtask 1 focused on classifying text as 'original' or 'fake' news and subtask 2 targeted to categorize texts into 'false', 'half true', 'mostly false', 'partly false' and 'mostly true'. for subtask 1, a system was developed to classify texts as fake or original using a corpus created by @xcite . the dataset included 5091 texts from youtube comments of varying lengths in the malayalam language. the training, validation, and test sets contained 3257, 815, and 1019 texts, respectively, divided into 'original' and 'fake' categories. 'original' texts comprise 14031 words, while fake texts contain 23198 words ( ## conclusion and limitations our study explored a diverse range of models for detecting and classifying fake news. through the investigation of four ml models, three dl models, and four transformer models, we gained valuable insights into their performance and effectiveness in these tasks. in subtask 1, m-bert outperformed other transformer models, including ml and dl models, with a macro f1 score of 0.84, but surprisingly, the lr model with tf-idf feature extraction came close to 0.82. in subtask 2, malayalam-bert outperformed the other ml, dl, and transformer models with a macro f1 score of 0.496. some dl and ml models came close to this result. cnn with fasttext feature extraction came close to it with a macro f1 score of 0.463. although the system demonstrated strong performance in detecting malayalam fake news, it faced a significant challenge in classifying multi-class fake news due to a potential data imbalance. to address this limitation, further research and strategies, such as advanced algorithms tailored for imbalanced datasets, are needed to enhance classification accuracy.