CUET _ NLP _ G ood F ellows@ D ravidian L ang T ech EACL 2024: A Transformer-Based Approach for Detecting Fake News in D ravidian Languages (2024)

Abstract: In this modern era, many people have been using Facebook and Twitter, leading to increased information sharing and communication. However, a considerable amount of information on these platforms is misleading or intentionally crafted to deceive users, which is often termed as fake news. A shared task on fake news detection in Malayalam organized by DravidianLangTech@EACL 2024 allowed us for addressing the challenge of distinguishing between original and fake news content in the Malayalam language. Our approach involves creating an intelligent framework to categorize text as either fake or original. We experimented with various machine learning models, including Logistic Regression, Decision Tree, Random Forest, Multinomial Naive Bayes, SVM, and SGD, and various deep learning models, including CNN, BiLSTM, and BiLSTM + Attention. We also explored Indic-BERT, MuRIL, XLM-R, and m-BERT for transformer-based approaches. Notably, our most successful model, m-BERT, achieved a macro F1 score of 0.85 and ranked 4th in the shared task. This research contributes to combating misinformation on social media news, offering an effective solution to classify content accurately.

## introduction in recent years, there has been an unprecedented surge in user participation on social media platforms such as facebook and twitter, as individuals increasingly utilize these platforms @xcite . the users engage in the exchange of information, communication, and the continuous monitoring of current events. conversely, a significant portion of the recent information disseminated on these platforms is inaccurately represented and, at times, deliberately crafted to misguide users. this content category is commonly identified as "fake news," encompassing any deceptive or false information presented as authentic news @xcite . the anonymity afforded to users on social media provides an opportunity for disseminators of fake news to manipulate people's beliefs, trust, and opinions by intentionally spreading false information. rumors and misinformation propagate swiftly, adversely affecting personal relationships and social connections. moreover, they have the potential to induce anxiety and emotional distress by fostering unfavorable perceptions, subjecting individuals to public scrutiny, and contributing to social isolation @xcite . moreover, current news often makes statements without confirmed evidence. to determine if these real-time claims are true, we heavily depend on how well they match information from other sources. the shared task @xcite organized by dravidianlangtech@eacl 2024 foot_0 provided us with an opportunity to address this significant challenge. this task aims to categorize a given social media text as either original or fake. the data sources include diverse social media platforms like twitter and facebook. the objective of this research work is to develop a system capable of discerning whether a news sample is original or fake. the key contributions of this endeavor are outlined below: ## related work the detection of fake news in low-resource languages, including code-mixed texts, is gaining increasing attention. researchers have investigated various techniques for identifying fake news using benchmarked corpora in low-resourced languages. in this section, we provide a concise overview of relevant studies in this domain. @xcite addressed the challenge of detecting fake news through three machine learning models (mnb, lr, and ensemble) trained on code-mixed malayalam text using term frequency -inverse document frequency (tf-idf). they achieved a notable macro f1-score of 0.831 and secured 3 rd rank in the "fake news detection in dravidian languages" shared task at dravidianlangtech@ranlp 2023. in response to the urgent need for robust defenses against machine-generated fake news, fung et al. (2021) created a benchmark dataset and identified fake news using cross-media consistency checking. their proposed methodology surpassed the state-of-the-art models and achieved up to a 16.8% gain in accuracy. rasel et al. (2022) constructed a comprehensive bangla fake news dataset and have employed various machine learning, deep neural networks, and transformer models. the best performing models, cnn, cnn-lstm, and bilstm, achieved notable accuracies of 95.9%, 95.5%, and 95.3%, respectively. li et al. (2021) outlined the system for the aaai 2021 shared task on covid-19 fake news detection in english, securing the 3 rd position with a weighted score of 0.9859 on the test set. they constructed an ensemble of pretrained language models, including bert, roberta, and ernie, and employed diverse training strategies like a warm-up, a learning rate schedule, and kfold cross-validation. shu et al. (2019) addressed the challenge of fake news detection on social media by introducing the trifn framework, a novel approach leveraging the inherent tri-relationship among publishers, news pieces, and users during dissemination. unlike traditional algorithms focusing solely on news content, trifn concurrently models publisher-news relations and user-news interactions. zhou and zafarani (2020) addressed the pressing issue of fake news, emphasizing its detrimental impact on democracy, justice, and public trust. evaluating detection methods from multiple perspectives, including false knowledge, writing style, propagation patterns, and source credibility, the survey encourages interdisciplinary research. sharif et al. (2021) presented a detailed description of a system developed for encompassing covid-19 fake news detection in english (task-a) and hostile post detection in hindi (task-b) using svm, cnn, bilstm, and cnn+bilstm with tf-idf and word2vec embedding. their system achieved notable results, with the highest weighted f1 score of 94.39% in task-a and 86.03% coarse-grained and 50.98% fine-grained f1 scores in task-b. ## task and dataset description the surge in online social media usage has revolutionized communication, enabling users to exchange information, engage in conversations, and stay informed about current events. however, this convenience has also led to the widespread dissemination of false information, commonly known as fake news, aiming to mislead users. this shared task @xcite focuses on classifying social media texts as either original or fake news. the dataset comprises of text samples collected from diverse social media platforms, including twitter and facebook. it is organized into two distinct classes: "fake" and "original". the following outlines the definitions of the classes: ## results in this section, we provide comparisons of the performance achieved by different machine learning, deep learning, and transformer-based methods. the performance evaluation of various classifiers for fake news detection showcases intriguing insights into their predictive capabilities. a detailed summary of the precision (p), recall (r), and macro-f1 (mf1) scores attained by each model on the test set is provided in transitioning to deep learning architectures, cnn, bilstm, and bilstm + attention exhibited competitive performances. while bilstm showed slightly higher precision (p), bilstm + attention demonstrated better recall (r), underscoring the importance of attention mechanisms for discerning subtle patterns. however, the standout performers were the transformer-based models, xlm-r and m-bert. outperforming other models in precision (p), recall (r), and macro-f1 (mf1) scores, m-bert emerged as the top performer with the highest scores across all metrics, achieving a macro-f1 (mf1) score of 0.85. ## conclusion in our study, we set out to tackle the task of classifying fake and original news. through a detailed comparison of various machine learning (ml), deep learning (dl), and transformer-based models, we found that m-bert delivered the most impressive performance, boasting a macro f1 score of 0.85, surpassing all other models. looking ahead, we plan to refine our approach further by exploring ensemble techniques in future research endeavors, aiming for an even more effective solution to combat misinformation.