CUET _ DUO @ D ravidian L ang T ech EACL 2024: Fake News Classification Using M alayalam- BERT (2024)

Abstract: Identifying between fake and original news in social media demands vigilant procedures. This paper introduces the significant shared task on ‘Fake News Detection in Dravidian Languages - DravidianLangTech@EACL 2024’. With a focus on the Malayalam language, this task is crucial in identifying social media posts as either fake or original news. The participating teams contribute immensely to this task through their varied strategies, employing methods ranging from conventional machine-learning techniques to advanced transformer-based models. Notably, the findings of this work highlight the effectiveness of the Malayalam-BERT model, demonstrating an impressive macro F1 score of 0.88 in distinguishing between fake and original news in Malayalam social media content, achieving a commendable rank of 1st among the participants.

## introduction a growing number of people are choosing to get their news from social media platforms rather than traditional news outlets in an era where online interactions are becoming more common. news consumption on social media differs from traditional media, such as newspapers and television, regarding timeliness and affordability. social media preference is also influenced by the ease with which news can be shared, commented on, and discussed with friends and other readers. however, the ease of sharing content via social media and the costeffectiveness of online news distribution have led to the massive spread of fake news. this trend is particularly pertinent in the context of our paper on fake news detection in the malayalam language. @xcite introduced ensemblebased models for detecting offensive language in mixed-script social media posts. many task formulations, datasets, and natural language processing (nlp) solutions have been used to investigate the intricacies of identifying fake news in the literature @xcite . this paper discusses the challenges of identifying fake news, focusing on the malayalam language. the aim is to curb the spread of misinformation in this language space by providing culturally sensitive insights and solutions by exploring methodologies. recent data on the impact of facebook referrals, which shows that 20% of traffic goes to reliable websites and 50% goes to fake news sites, underscores the pressing nature of the problem. considering that 62% of american adults get their news from social media, recognizing and mitigating the influence of fake content in online sources is more crucial than ever @xcite . this research contributes to the domain of fake news detection in the malayalam language through the following key aspects: sivanaiah et al. (2022) prepared fake news datasets for several low-resource languages and applied logistic regression and bert models for fake news classification. it was demonstrated through rigorous experiments that 'bert-based-multilingualcased' achieved a maximum f1 score of around 98%. at the same time, logistic regression reached approximately 95% in low-resource indian languages such as malayalam, gujarati, and tamil. hariharan and anand kumar (2022) created a multilingual low-resource fake news classification dataset and examined the impact of transformerbased models, such as multilingual bert, xlm-roberta, and muril. for telugu, kannada, tamil, and malayalam, they assessed four transformer models: mbert, xlm-roberta, in-dicbert, and muril. however, raja et al. 2022 demonstrated that, for these low-resource languages, muril had a higher accuracy in identifying fake news. the dravidianlangtech@ranlp @xcite 2023 session "fake news detection in dravidian languages" concentrated on malayalam content. in particular, the xlmroberta-based model performed exceptionally well, obtaining a macro f1-score of 0.90. in @xcite proposed transformer models such as m-bert, albert, bert, and xlnet. m-bert outperformed competitors with a robust f1 score of 0.74, surpassing xlnet and albert, which achieved accuracy scores of 0.71 and 0.66, respectively. using transformer-based models for language analysis, the study @xcite explored the nuances of identifying fake news. the mural-base-cased version of muril was refined using a dravidian languagecurated dataset. @xcite addressed the scarcity of resources for the bangla language in fake news detection by constructing a dataset of 4678 distinct news instances. employing various machine learning, deep neural network, and transformer models, including cnn, cnn-lstm, and bil-stm, they achieved state-of-the-art accuracy ranging from 95.3% to 95.9%, showcasing notable improvements in accuracy and recall compared to previous studies when tested on both newly collected and existing datasets. rahman et al. ( 2022 ) created the bfnc dataset containing 5,000 instances of fake news and presented the fand-x framework using transformer-based and neural network-based techniques. with a maximum f1-score of 98% on the test data, experimental results showed that xlm-r outperformed other methods, demonstrating its efficacy in detecting fake news. @xcite conducted a comparison of various bilstm models for detecting false information. this research demonstrates the efficacy of sequential models in text classification and identifying deceptive information, suggesting a potential future evaluation compared to bert. in a similar application, the artificial intelligence initiative by facebook (kurasinski and mihailescu, 2020) incorporates bert as an integral component of its machine-learning strategy for detecting hate speech. in the past few years, pre-established models leveraging the transformer architecture introduced @xcite have gained prominence and serve a crucial function in sequence encoding and decoding. the effectiveness of these models in generating condensed contextualized embeddings for diverse texts inspired us to develop a system for detecting deceptive information based on these models. ## task and dataset descriptions for the goal of fake news identification in the malayalam language, the organizers created an almost balanced and standardized dataset. the primary purpose is to design a system that appropriately differentiates between fake and original news from social media posts in malayalam. the dataset utilized in this challenge is derived from the corpus given by the workshop organizers @xcite . the work entails sorting social media statements into two predetermined classes: fake and original news. ## methodology the proposed method experimented with several machine learning (ml), dl, and transformer-based baselines with fine-tuning the hyperparameters. figure 1 demonstrates a schematic process of the employed models. ## error analysis the fake news detection performance of the malayalam-bert model in malayalam exhibits outstanding performance, especially evident in the high true positive count. figure 2 shows the confusion matrix of the best-performed model (malayalam-bert) that highlighted the finest accuracy achieved by correctly tagging 442 out of 512 fake samples, demonstrating an effective capability to detect fake samples. however, a limitation arises in precision since 70 original samples were incorrectly identified as fake, indicating a vulnerability to false news detection. less effective data preparation techniques could cause the misclassification. addressing this issue through better preprocessing approaches may enhance precision and contribute to a more accurate classification. this observed pattern necessitates careful analysis and adjustment of the model's discriminatory capabilities. further evaluations of the validation and test sets are essential to thoroughly examine the model's adaptability. figure 3 illustrates some predicted outcomes by the best-performed model (malayalam-bert). ## conclusion this work addresses the challenges of fake news detection in malayalam by exploiting three ml, three dl, and two transformer-based models. experimental investigations on the test dataset revealed that malayalam-bert demonstrated superior performance among all models, achieving the highest macro f1 score (0.88). this finding highlights the proficiency of transformer-based strategies, precisely the efficiency of the malayalam-bert architecture, in excelling at the challenge of fake news identification. for future improvements, employing more language-specific preprocessing techniques and exploring ensemble models could enhance the overall performance of fake news detection in malayalam. these strategies may contribute to refining the accuracy and robustness of the models in identifying misinformation effectively.