Habesha@ D ravidian L ang T ech 2024: Detecting Fake News Detection in D ravidian Languages using Deep Learning (2024)

Abstract: This research tackles the issue of fake news by utilizing the RNN-LSTM deep learning method with optimized hyperparameters identified through grid search. The model’s performance in multi-label classification is hindered by unbalanced data, despite its success in binary classification. We achieved a score of 0.82 in the binary classification task, whereas in the multi-class task, the score was 0.32. We suggest incorporating data balancing techniques for researchers who aim to further this task, aiming to improve results in managing a variety of information.

## introduction getting updates about what's happening in the world is important for us. it helps us learn more about the different things going on. many people like to watch the news or read the newspaper in the morning with a cup of tea to stay informed. fake news is one of the most significant scourges in our digitally interconnected world. it is broadly defined as a subject encompassing news, data, reports, and information that is either wholly or partially false @xcite . if the news is not true, it can confuse people and spread false information. sometimes, fake news is used to create rumors or harm the reputation of political leaders. to tackle this issue, we've suggested a system to identify fake news. however, with the enormous amount of data on the internet and social media, it's challenging to manually check if each piece of news is fake or not @xcite . this pervasive phenomenon acts as a wildfire, influencing countless individuals daily. the reach of fake news is extensive and can pose a significant threat to a nation's security, economy, prosperity, and the well-being of its citizens. unfortunately, many people may not be fully aware of how profoundly fake news can affect the matters that sur-round them, and they may lack the necessary skills to discern and handle such situations when they arise @xcite @xcite . fake news is crafted with the explicit intent of disseminating information under the guise of propaganda or a hoax, ultimately aimed at achieving financial or political gains @xcite . this deceptive practice manipulates public opinion, steering it towards falsehoods and distortions. the creators of fake news strategically weave narratives designed to exploit vulnerabilities and biases, with the ultimate goal of influencing individuals and, in turn, shaping societal perceptions. this insidious tactic not only undermines the credibility of information but also poses a significant threat to the foundations of democracy and the well-being of communities. recognizing the nefarious intentions behind fake news is essential for fostering a more discerning public and cultivating a media landscape rooted in truth and integrity @xcite . the organizers of a shared task organized this study, which consists of two tasks: identifying multiple labels in malayalam news and classifying a given social media text as either original or fake @xcite . the organization of this paper is as follows: in section 2, we thoroughly examine several related research studies. section 3 focuses on explaining the tasks we are considering, while section 4 gives a detailed discussion of the chosen methodology and experiments carried out for the task. section 5 is where we present results and have discussions about them. lastly, in section 6, we draw conclusions and discuss possible future trends in research within this field. ## related works numerous researchers have delved into the study of fake news detection, employing a variety of ap-proaches to unravel the complexities of understanding and categorizing fake news. notably, these approaches encompass machine learning, deep learning, and transformer-based methodologies. below, we explore a few notable studies that showcase the diversity of techniques employed: in earlier studies, researchers leveraged on the traditional machine learning techniques @xcite @xcite @xcite . @xcite introduces a model for fake news detection utilizing n-gram analysis and machine learning methodologies. the study delves into an examination and comparison of two distinct feature extraction techniques and six varied machine classification methods. the experimental evaluation reveals optimal performance when employing term frequency-inverted document frequency (tf-idf) as the feature extraction technique and linear support vector machine (lsvm) as the classifier, achieving an impressive accuracy rate of 92%. @xcite employed a straightforward method for fake news detection, utilizing a naive bayes classifier. this approach was translated into a software system and evaluated against a dataset comprising facebook news posts. the achieved classification accuracy on the test set reached approximately 74%, a commendable result given the relatively uncomplicated nature of the model. the article discusses various avenues for potential improvement, indicating that the obtained results offer insights into addressing the fake news detection problem using artificial intelligence methods. @xcite introduces an intelligent detection system for news classification, addressing both real and fake news tasks, utilizing an ensemble voting classifier. the approach involves the incorporation of eleven well-established machine-learning algorithms, including naïve bayes, k-nn, svm, random forest, artificial neural network, logistic regression, gradient boosting, ada boosting, among others. through cross-validation, the top three performing machine-learning algorithms are selected for integration into the ensemble voting classifier. the experimental results validate the efficacy of the proposed framework, achieving an impressive accuracy rate of approximately 94.5%. additionally, other key metrics such as roc score, precision, recall, and f1 demonstrate outstanding performance. the proposed detection framework not only attains high accuracy but also effectively identifies crucial features within news data. these identified features hold promise for implementation in other classification techniques, extending the utility of the system to detect fake profiles and messages. with the advent of deep learning, researchers explored the application of neural networks, @xcite @xcite @xcite . these deep learning models demonstrated improved performance in capturing sequential dependencies within textual data. to tackle the issue of fake news, the @xcite propose a neural network architecture designed to precisely predict the stance between a provided pair of headline and article body. their model surpasses the performance of existing architectures by 2.5%, achieving an impressive accuracy rate of 94.21% on the test data. the authors sahoo and gupta (2021) present an innovative approach to automatic fake news detection within the chrome environment, enabling the detection of fake news on facebook. their methodology involves the utilization of various features linked to a facebook account, coupled with certain news content features, to analyze account behavior through deep learning techniques. through experimental analysis on real-world data, the authors demonstrate that their proposed fake news detection approach attains higher accuracy compared to existing state-of-the-art techniques. in response to the proliferation of fake news, the imperative for computational methods to detect them has become increasingly apparent. the primary objective of fake news detection is to empower users to discern various forms of fabricated information. the determination of the news veracity hinges on a decision-making process influenced by previously encountered instances of fake or authentic news. various models can be employed to discern deceptive news circulating on social media platforms. kaliyar (2018) makes a two fold contribution to the field. firstly, the authors introduce datasets encompassing both fake and real news, undertaking diverse experiments to design effective fake news detectors. leveraging natural language processing, machine learning, and deep learning techniques, they classify the datasets, providing a comprehensive assessment of fake news detection. their contribution extends to encompass fake news categorization, incorporating existing algorithms derived from machine learning techniques. in this research chauhan and palivela (2021), a profound exploration into distinguishing false news from authentic sources is conducted through a deep learning-based approach. the cornerstone of the proposed model is a lstm neural network. complementing the neural network, a glove word embedding is employed to represent textual words as vectors. additionally, tokenization is utilized for feature extraction or vectorization, enhancing the model's capacity. the integration of n-grams further refines the proposed approach. a comprehensive comparative analysis of multiple fake news detection techniques is undertaken. the results of the proposed model are meticulously evaluated using accuracy metrics, revealing an exceptional performance with an accuracy rate of 99.88%. this underscores the effectiveness of the lstm-based model and its ability to discern false news with an exceedingly high level of accuracy. ## description of tasks and dataset there is an escalating demand for the detection of fake news within social media texts. the dataset for this crucial task has been generously provided by the organizers of the fake news detection in dravidian languages-dravidianlangtech@eacl 2024 @xcite and our team, habesha, actively participated in this shared endeavor. throughout our involvement in this collaborative initiative, we immersed ourselves in two distinctive tasks. the primary task aimed to classify social media texts as either original or fake. the data sources encompassed various social media platforms such as twitter and facebook. given a social media post, the shared task mandated the classification of the content as either fake or authentic news. in the second task, namely the fake news detection from malayalam news (fakedetect-malayalam) shared task, researchers were provided with a platform to address the formidable challenge of identifying and flagging fake news within the realm of malayalam-language news articles. accurate misinformation detection is crucial for fostering trustworthy communication in an era of information overload. the core objective of the fakedetect-malayalam shared task was to inspire participants to develop effective models capable of accurately detecting and categorizing fake news articles in the malayalam language into different categories. in this context, we considered five fake categories -false, half true, mostly false, partly false, and mostly true. through our engagement with these two tasks, our goal was to contribute to a comprehensive understanding of fake news detection, with a specific emphasis on addressing the nuances associated with youtube comments in both tasks. we utilized a total of 4,072 data points for training in task 1, which involves classifying fake and original content. additionally, 1,019 data points were allocated for evaluating the model's performance. in task 2, focusing on fake news detection from malayalam news, we employed 1,669 data points for training and reserved 250 data points for evaluation purposes. for a more comprehensive overview of the statistics related to these specific datasets, kindly refer to figure 1 for task 1 and figure 2 for task 2. figure 2: task 2 data set statistics in figure 1 , we observe an unbalanced distribution within the two-class dataset, indicating an uneven representation of the classes. in the meantime, figure 2 shows that task 2 is a multi-label assignment with five classes, which shows how the data is spread out in a very different way. this imbalance poses a significant challenge, potentially resulting in biased predictions and impacting the overall performance of the trained model. recognizing and addressing such imbalances becomes imperative for fostering a more robust and unbiased predictive model @xcite @xcite . ## methodology this section offers an in-depth insight into the methodology applied in this study, with a specific emphasis on data preprocessing and the adoption of a deep learning approach for fake news detection. our primary goal was to perform fake news detection for both tasks. initiating our analysis, we acknowledged the significance of meticulous data preprocessing. this critical step involved purifying and formatting the raw data to optimize its appropriateness for subsequent stages in our model. ## results and discussion as outlined in section 4.1, we implemented the recurrent neural network (rnn) model for both shared tasks. our outcomes revealed a macro-f1 score of 0.82 for task 1 and 0.32 for task 2. in task one, our model exhibited promising performance by effectively distinguishing between fake and original news, resulting in a superior outcome compared to task two. this discrepancy in performance can be attributed to the nature of task two, involving multi-label classification with an unbalanced data set, which presented challenges for achieving satisfactory results. the inherent complexity of addressing multiple classes in task 2 posed difficulties, leading to suboptimal performance in contrast to the binary classification nature of task 1. ## conclusion and future work the dissemination of fake news or misinformation poses a significant challenge, steering information in undesired directions and impeding the acquisition of reliable and timely information. to address this issue, we employed the deep learning approach, specifically rnn-lstm. for optimal model training, we utilized grid search methods to fine-tune hyper parameters. in binary classification, our approach yielded favorable results, showcasing its effectiveness in distinguishing between genuine and fake information. however, the application of multi-label classification suffered from the presence of unbalanced data, which led to less than ideal results. as a recommendation for researchers aiming to extend this task, we suggest incorporating data balancing techniques. by addressing the imbalance in the dataset, more robust and accurate results can be achieved in the context of multi-label classification, enhancing the model's ability to handle diverse and nuanced information effectively.