CUET _ B inary_ H ackers@ D ravidian L ang T ech EACL 2024: Fake News Detection in M alayalam Language Leveraging Fine-tuned M u RIL BERT (2024)

Abstract: Due to technological advancements, various methods have emerged for disseminating news to the masses. The pervasive reach of news, however, has given rise to a significant concern: the proliferation of fake news. In response to this challenge, a shared task in Dravidian- LangTech EACL2024 was initiated to detect fake news and classify its types in the Malayalam language. The shared task consisted of two sub-tasks. Task 1 focused on a binary classification problem, determining whether a piece of news is fake or not. Whereas task 2 delved into a multi-class classification problem, categorizing news into five distinct levels. Our approach involved the exploration of various machine learning (RF, SVM, XGBoost, Ensemble), deep learning (BiLSTM, CNN), and transformer-based models (MuRIL, Indic- SBERT, m-BERT, XLM-R, Distil-BERT) by emphasizing parameter tuning to enhance overall model performance. As a result, we introduce a fine-tuned MuRIL model that leverages parameter tuning, achieving notable success with an F1-score of 0.86 in task 1 and 0.5191 in task 2. This successful implementation led to our system securing the 3rd position in task 1 and the 1st position in task 2. The source code will be found in the GitHub repository at this link: https://github.com/Salman1804102/ DravidianLangTech-EACL-2024-FakeNews.

## introduction social media has gradually become an integral part of our lives, with regular posting and commenting being commonplace. unfortunately, this platform is often misused, as individuals spread rumors by purposefully posting fake news to attack others and cause harm @xcite . given the importance of accurate information, it becomes crucial to curb the pervasiveness of fake news for the greater good. the widespread dissemination of false information carries catastrophic implications and potential dangers, particularly in the political and social spheres @xcite . a statistical analysis of the american public's aptitude for discerning between authentic and false news indicates a troubling pattern, as only 26% of survey participants express a high level of confidence in their ability to make this distinction @xcite . this low number underscores the urgent need for an automated system to detect fake news. numerous studies have been established for detecting fake news in high-resourced languages like english, arabic, spanish, french, german, etc. @xcite @xcite @xcite . but there is still much work to be done, especially for low-resourced languages like malayalam, particularly in codemixed text @xcite . besides, in contrast to other dravidian languages, malayalam presents unique linguistic intricacies, encompassing dialect variations, nuanced word semantics, idiomatic expressions, and more @xcite . these intricacies pose challenges in processing and analyzing malayalam text. this shared task addresses precisely this issue, aiming to develop an automated system for detecting fake news and classifying its severity by categorizing news into various types. as participants in this shared task, the contributions of this paper are outlined as follows: the rest of the paper follows this structure: section 3 presents the task and dataset description, section 4 discusses the methodology, section 5 covers the results and error analysis, and section 6 describes the conclusion and outlines future work. ## literature review the surge in fake news incidents prompted extensive research, initially relying on machine learning for detection. a specific study @xcite harnessed machine learning and n-gram analysis, achieving a no however, the shift from deep learning to transformer-based models like bert has notably improved fake news detection accuracy, especially in code-mixed-text contexts @xcite . a study @xcite on a bengali fake news dataset revealed xlm-r's superior accuracy of 98%. whereas in the multilingual fake news classification scheme, hariharan and anand @xcite focused on lowresourced tamil and malayalam. they assessed the effectiveness of transformer-based models like m-bert, xlm-r, and muril. the focus on the low-resourced dravidian languages has continuously increased. @xcite achieved an impressive f1-score of approximately 95% utilizing lr and 98% utilizing bert models in one 1 https://dravidianlangtech.github.io/2023/ of the fake news detection endeavors for several indian low-resourced languages like tamil, kannada, gujarati, and malayalam. the author @xcite in this study delved into the use of a dataset sourced from youtube comments featuring malayalam-english code-mixed text. the study explored the effectiveness of camem-bert, distil-bert, electra, and xlm-r models in this context. remarkably, electra achieved an impressive f1-score of 99.33%. in another dra-vidianlangtech@ranlp 2023 study, @xcite implemented the muril base variant model and achieved a notable f1-score of 87% for malayalam code-mixed text. within the context of fake news detection, addressing multi-class classification scenarios where news articles encompass varying degrees of truthfulness becomes crucial @xcite . a recent study @xcite has delved into this challenge, addressing the multiclass classification of fake news with labels such as 'true', 'partially false', 'false', and others. to handle class imbalance, the researchers experimented with chatgpt-based data augmentation, achieving an f1-score of 23% with m-bert proving to be the most effective. ## task and dataset description this shared task @xcite on 'fake news detection in dravidian languages' consisted of two separate sub-tasks. in task 1, participants aimed to distinguish whether a post or comment is 'original' or 'fake'. task 2 involved a more nuanced challenge, requiring participants to categorize news into five distinct labels: 'false' (f), 'mostly false' (mf), 'partly false' (pf), 'mostly true' (mt), and 'half true' (ht). the competition organizers provided a dataset @xcite in multilingual and codemixed malayalam for these tasks. task 1 comprised three distinct datasets (train, dev, and test), while task 2 involved two separate datasets (train and test). the training dataset in task 1 demonstrated a near balance, but task 2's training dataset exhibited a significant imbalance, with only a single occurrence of the mt class. the 'false' class constituted about three-fourths of the samples. however, task 2 lacked a dedicated dev set. further dataset statistics are detailed in ## methodology this section outlines the model framework devised to tackle the issue detailed in section 2. initially, to preprocess the text, we conducted several steps including the removal of emoticons, pictographs, urls, and brackets. following this, we employed feature extraction techniques to retrieve essential information. our choice of feature extraction techniques was driven by the linguistic nuances inherent in such languages. tf-idf @xcite was selected for the ml models to effectively weigh term importance based on frequency, aligning with the need for interpretability in the context of fake news detection. on the other hand, glove embeddings @xcite were employed for the dl models, as they excel in capturing semantic relationships and contextual nuances within the text. this choice aimed to enhance the models' understanding of the intricacies present in malayalam, contributing to more effective fake news classification. the overview of the proposed methodology is shown in figure 1 . ## results and error analysis this section delves into the results and error analysis of the proposed fake news detection system. a detailed performance analysis is shown in ## conclusion this paper introduces a fake news detection system tailored for code-mixed malayalam. it encompasses diverse models including ml, dl, and transformer-based models. through extensive experimentation, evaluation, fine-tuning, and hyperparameter adjustments, the system showcases promising outcomes. across both tasks, muril emerges as the top performer, demonstrating its superior ability to handle code-mixed and transliterated malayalam. the system achieves noteworthy f1-scores of 0.86 and 0.519, securing the 3 rd and 1 st positions in task 1 and task 2, respectively. in the future, the exploration of fake news detection in other low-resourced dravidian languages could be a worthwhile pursuit. implementing data augmentation instead of relying solely on class weight adjustments for managing highly imbalanced datasets might prove more effective. additionally, the utilization of hybrid models, combining transformers and dl models, holds the potential to yield improved results. furthermore, exploring ensembles of transformer-based models could lead to superior performance.