T ech W hiz@ D ravidian L ang T ech 2024: Fake News Detection Using Deep Learning Models (2024)

Abstract: The ever-evolving landscape of online social media has initiated a transformative phase in communication, presenting unprecedented opportunities alongside inherent challenges. The pervasive issue of false information, commonly termed fake news, has emerged as a significant concern within these dynamic platforms. This study delves into the domain of Fake News Detection, with a specific focus on Malayalam. Utilizing advanced transformer models like mBERT, ALBERT, and XMLRoBERTa, our research proficiently classifies social media text into original or fake categories. Notably, our proposed model achieved commendable results, securing a rank of 3 in Task 1 with macro F1 scores of 0.84 using mBERT, 0.56 using ALBERT, and 0.84 using XMLRoBERTa. In Task 2, the XMLRoBERTa model excelled with a rank of 12, attaining a macro F1 score of 0.21, while mBERT and BERT achieved scores of 0.16 and 0.11, respectively. This research aims to develop robust systems capable of discerning authentic from deceptive content, a crucial endeavor in maintaining information reliability on social media platforms amid the rampant spread of misinformation.

## introduction navigating the digital realm, social media emerges as a pivotal force in disseminating information, presenting both opportunities and challenges. the surge in fake news instigated the initiation of the fake news detection in dravidian languages task @xcite , slated for presentation at dravidianlangtech@eacl in 2024 foot_0 . task 1 plays a crucial role, concentrating on categorizing social media text to meticulously distinguish between original and fake news. the challenge lies in discerning genuine content from misleading information. task 2 shifts focus to detecting fake news in malayalam-language articles, aiming to develop models classifying misinformation into five distinct types. beyond bolstering natural language processing capabilities, the initiative actively contributes to fostering a more informed and reliable digital environment. this effort not only upholds information integrity but also reinforces the pillars of a trustworthy digital space, marking a beacon for responsible technological advancements and ethical digital communication practices. ## related works different deep learning models, including cnns, lstms, ensembles, and attention mechanisms had been employed for fake news detection by kum @xcite . the cnn + bidirectional lstm ensembled network with attention had achieved the highest accuracy. @xcite had proposed "exbake", a model for detecting fake news that uses bidirectional encoder representations from transformers (bert). the model had outperformed previous models in terms of f1-score by effectively analysing the relationship between news headlines and body text. @xcite had addressed early detection challenges and labelled data scarcity in fake news. the proposed transformerbased framework had taken into account both news content and social contexts, resulting in greater accuracy shortly after news dissemination. they had emphasised the significance of incorporating multiple features for better classification. @xcite had made a contribution to the detection of disinformation by presenting a content-based classification approach based on pre-trained transformer models. their experiments with models such as xlnet, bert, and roberta had yielded promising results, highlighting the effectiveness of transformers in achieving high accuracy even with small datasets. the transformer-based fake news detection framework proposed by @xcite had demonstrated higher accuracy in early detec application of cross lingual model for sentiment classification on tamil-english code mixed data had been proposed by @xcite . @xcite had employed an attentionbased transformer model for social media fake news detection, discovering a 15 percentage improvement with a hybrid cnn model, underscoring the significance of attention mechanisms. @xcite had explored transformer-based neural network models, showcasing effectiveness in fake news detection using precision, f1-score, and recall metrics. @xcite proposed an xlnet fine-tuning model, outperforming existing models in multi-class and binary-class fake news detection. the katmf framework by @xcite , integrating knowledge augmented transformer, had excelled in multimodal fake news detection on a real-world dataset. transfake, a multi-task transformer model introduced by @xcite had outperformed competitors by jointly modeling body content and comments, emphasizing the importance of considering multiple modalities for comprehensive fake news analysis. ## data set the datasets utilized for implementing fake news detection in task 1 comprised the training, evaluation, and test datasets provided by the shared task organizers @xcite . each instance in the training dataset was labeled to specify whether the text is fake or original. for task 2, the datasets used for fake news detection included the training and test datasets from the task organizers. each instance in the task 2 training dataset was labeled to indicate one of the five fake categories: false, half true, mostly false, partly false, and mostly true. ## system description in task 1, social media text was classified as original or fake using xlm-roberta, albert, and mbert. the methodology involved preprocessing data by removing unwanted characters and training models on datasets from the task organizers. xlm-roberta, with the highest development accuracy, was chosen for final predictions. task 2 addressed the fakedetect-malayalam challenge, focusing on five fake news categories. transformer models (xlm-roberta, mbert, and bert) handled data encoding and tokenization. in the absence of a designated development dataset, training data was split for validation. optimization with adamw and cross-entropy loss during training led to mbert achieving the highest accuracy, highlighting the importance of precise classification in streamlined preprocessing, training, and model selection. figure 1 illustrates the proposed architecture for both tasks. ## result the evaluation of task performance focused on the macro-f1 score. in task 1 of dravidian-langtech@eacl 2024, fake news detection assessment relied on macro f1-score and accuracy. xlmroberta secured 3rd position on the leaderboard with a notable macro f1-score of 0.84 and accuracy of 0.84. despite similar task 1 scores, xlmroberta's overall excellence, especially in cross-lingual tasks, pre-training, and adaptability, influenced its preference over mbert. models like xlmroberta, albert, and mbert made unique contributions, as depicted in ## error analysis the f1 score from the xlmroberta model for both task 1 and task 2 exposes false positive and false negative prevalence, linked to inherent data imbalance. class "false" with more instances, exhibits a high f1 score, precision, and recall, indicating fake news. lower training instances intensify misclassifications, highlighting the data imbalance impact. to address this, data augmentation is suggested for enhanced model performance. misclassified malayalam texts due to data imbalance are detailed in ## limitation xlmroberta, albert, and mbert excel but face hurdles: cross-lingual challenges, biased predictions due to data imbalances, uncertainties in generalizing to new datasets, resource-intensive fine-tuning for diverse languages, interpretability issues, and a textual focus overlooking multimodal fake news complexities. recognizing and addressing these constraints is essential for refining approaches and ensuring robust fake news detection in transformer-based models. ## conclusion the dravidianlangtech@eacl 2024 initiative, focused on fake news detection in dravidian languages, signifies a critical step in countering misinformation on social media. task 1, employing mbert, albert, and xmlroberta, revealed xmlroberta as the optimal model, showcasing its prowess in discerning authenticity. in task 2, where xmlroberta, mbert, and bert were employed, xmlroberta consistently outperformed, emphasizing its effectiveness in classifying malayalam-language articles. this collective effort not only advances natural language processing but also reinforces the importance of reliable information dissemination in the digital age. the xmlroberta model's dual-task success highlights its pivotal role in navigating the challenges of fake news, setting the stage for continued innovation in dravidian language technology.