P unny_ P unctuators@ D ravidian L ang T ech- EACL 2024: Transformer-based Approach for Detection and Classification of Fake News in M alayalam Social Media Text (2024)

Abstract: The alarming rise of fake news on social media poses a significant threat to public discourse and decision-making. While automatic detection of fake news offers a promising solution, research in low-resource languages like Malayalam often falls behind due to limited data and tools. This paper presents the participation of team Punny_Punctuators in the Fake News Detection in Dravidian Languages shared task at DravidianLangTech@EACL 2024, addressing this gap. The shared task focuses on two sub-tasks: 1. classifying social media texts as original or fake, and 2. categorizing fake news into 5 categories. We experimented with various machine learning (ML), deep learning (DL) and transformer-based models as well as processing techniques such as transliteration. Malayalam-BERT achieved the best performance on both sub-tasks, which obtained us 2^{nd} place with a macro f_1 -score of 0.87 for the subtask-1 and 11^{th} place with a macro f_1 -score of 0.17 for the subtask-2. Our results highlight the potential of transformer models for low-resource languages in fake news detection and pave the way for further research in this crucial area.

## introduction in the current digital era, fake news and the spread of false information are widespread issues that have negative effects on people, communities, and countries @xcite . people can be misled and deceived by fake news, which can cause them to lose trust in organizations and information sources. the dissemination of misleading narratives and the promotion of biased opinions can cause societal division and conflict @xcite . therefore, using cutting-edge natural language processing (nlp) techniques, academics, policymakers, and stakeholders are working to construct strong * authors have contributed equally to this work computational systems to prevent the dissemination of fake content @xcite . researchers have actively pursued the development of effective solutions to detect fake news across multiple languages in recent years @xcite . fake news detection systems predominantly focus on high-resource languages like spanish and english, neglecting lowresource dravidian languages like tulu, malayalam, tamil, telugu, and kannada due to resource scarcity @xcite . despite the widespread use of the malayalam language in kerala, there is a lack of research on this language, necessitating the development of robust models for detecting fake news in malayalam @xcite . malayalam's unique linguistic complexities, such as dialect variations, word semantics, and idiomatic expressions, make it challenging to process and analyze its text @xcite . this work aims to develop a classification system for detecting fake news in malayalam using various language technologies, aiming to identify fake articles written in the malayalam language accurately. to effectively address the challenge, this work's major contributions are demonstrated by the following: ## related work the widespread use of social media and accessible internet access has resulted in the creation of mil-lions of posts and comments per minute @xcite . fake news on social media leads to wrong judgments, prompting studies on various machine learning and deep learning models using different word embedding techniques @xcite . the best score f 1 -score they obtained using svm was 94.39% in task-a. rasel et al., 2022 achieved 95.9% accuracy by building a dataset with 4678 distinct news and improved the existing dataset accuracy from 1.4% to 3.4% using cnn. roy et al., 2019 developed cnn and bilstm networks individually, then fed them into a multi-layer perceptron model (mlp) which resulted in 44.87% accuracy. rai et al., 2022 proposed a bert model with a feed-forward network with 768 hidden sizes connected to an lstm layer for fake news classification, outperforming the vanilla pre-trained model on two datasets with a 2.50% and 1.10% increase in accuracy. research primarily focuses on high-resource languages, with few studies on detecting abusive language in lowresource languages like malayalam. coelho et al., 2023 achieved 0.831 macro f 1 -score through applying tf-idf as a feature extraction technique and ensembling three machine learning models (mnb, lr, svm) with majority voting. bala and krishnamurthy, 2023 fine-tuned the muril variant named "mural-base-cased" in detecting fake news in dravidian languages resulting accuracy of 87%. wani et al., 2023 in their work, identify toxic fake news to save time on assessing non-toxic instances. traditional and transformerbased machine learning techniques were employed and the linear svm method outperformed bert svm, rf, and bert rf with an accuracy of 92%. using a newly annotated dataset, another study (chakravarthi et al., 2023) showed muril as a multilingual transformer by effectively detecting abusive comments in the low-resource tamil language. several machine learning, deep neural networks, and transformer-based approaches were utilezed by rahman et al., 2022 to analyze 5k fake news data, achieving a maximum f 1 -score of 98% using xlm-r. ## task and dataset descriptions for shared task-1, the task organizer foot_0 created a benchmark corpus for fake news detection @xcite . we utilized the corpus provided by the shared task-1 organizer to create the fake news detection classifier model. there are two subtasks in this shared task: subtask-1 and subtask-2. the goal of subtask-1 is to determine if a social media text is original or fake in the malayalam language. in subtask-2, the objective is to create efficient models that can precisely identify and categorize fake news articles in malayalam into five distinct classes. this subtask-2 offers five classes, including false, half true, mostly false, partly false, and mostly true. for subtask-1, the train, valid, and test splits of this dataset comprise 3257, 815 and 1019 texts respectively. class-wise samples and dataset statistics are provided in for subtask 2, the train and test splits of this dataset comprise 1669 and 250 texts. in the pre-processing stage, we filter out urls, emojis, and assorted symbols present within the provided dataset. ployed for developing the baseline models. performance of the classification of both tasks improved significantly by fine-tuning the transformerbased models. to identify fake news, this work employed four pre-trained transformer-based models: xlm-r, malayalam bert, mbert, and indic-bert. to be more specific, we fine-tuned the huggingface transformer 2 library's "xlm-roberta-base" @xcite , "l3cubepune / malayalam bert" @xcite , "bertbase-multilingual-cased" @xcite and "indic-bert" @xcite . figure 1 depicts the developed system's schematic process. ## results and analysis this section evaluates the performance of different models in recognizing fake news detection in dravidian languages. ## limitations our models depend on accurately turning words from one language into another (transliteration). even small mistakes can significantly impact the information, affecting how well our models work. furthermore, in subtask-2, the data was quite imbalanced, and no methods were applied to balance the data. future exploration of methods with multiple transliteration options, as well as augmentation for balancing the dataset can be investigated to further enhance the accuracy of our approach. ## conclusion this work aimed to detect and classify fake news from malayalam social media text. we have thoroughly investigated several machine learning (ml), and deep learning (dl) and transformer-based models for malayalam fake news identification and classification. the malayalam-bert model has proven to be more effective than the others, as evidenced by its highest macro f1-score of 0.87 for subtask-1 and 0.17 for subtask-2. in subtask-1, the model excels, securing the 2 nd position with a noteworthy macro f 1 -score of 0.87. in contrast, in subtask-2, it ranks 11 th with a macro f 1 -score of 0.17. to improve model performance in the future, we want to look at different architectures and use ensemble techniques. we'll explore different ways to tackle problems that come from imbalanced datasets.