From Dataset to Detection: A Comprehensive Approach to Combating M alayalam Fake News (2024)

Abstract: Identifying fake news hidden as real news is crucial to fight misinformation and ensure reliable information, especially in resource-scarce languages like Malayalam. To recognize the unique challenges of fake news in languages like Malayalam, we present a dataset curated specifically for classifying fake news in Malayalam. This fake news is categorized based on the degree of misinformation, marking the first of its kind in this language. Further, we propose baseline models employing multilingual BERT and diverse machine learning classifiers. Our findings indicate that logistic regression trained on LaBSE features demonstrates promising initial performance with an F1 score of 0.3393. However, addressing the significant data imbalance remains essential for further improvement in model accuracy.

## introduction detecting fake news is critical to identifying false or intentionally misleading information presented as legitimate news. in today's digital age, numerous websites spread fake news, significantly influencing society. the deceptive strategies employed by fake news to appear true further complicate this problem. fake news has far-reaching consequences, shaping public opinion, interfering with democratic processes like elections, and even inciting violence. researchers from various disciplines recognize the significance of studying and addressing this issue @xcite @xcite . although technology and social media positively impact society, they are not without limitations or defects. the spread of fake news and the threat of inaccurate data have grown, potentially leading to serious social problems. the effects of fake news can be wide-ranging, from being merely annoying to influencing and misleading entire communities or even countries. inaccurate information has a negative impact on society, according to related research. there are many ways to identify false news, including topic-agnostic, knowledge-based, machine-learning-based, and hybrid techniques. the importance of classifying fake news in malayalam and other low-resource languages lies in reducing the spread of false information and promoting informed decision-making in linguistically diverse societies. this requires developing effective models for classifying fake news in various languages, especially those with limited linguistic resources, such as malayalam. fake news frequently exploits linguistic and cultural particulars in low-resource languages, requiring the development of language-specific detection methods. effective models for identifying false information in languages with limited resources, as demonstrated in @xcite , @xcite , and @xcite , can play a crucial role in nurturing acceptance of diverse perspectives. keeping fake news detection in low-resource languages relevant and effective in the digital age. this paper makes a significant contribution to the field by providing a dataset and baseline machine learning models designed for classifying fake news into different classes based on the degree of misinformation it contains. to the best of our knowledge, this is the first dataset in this domain focusing on combating the spread of fake news to protect society from these inhumane acts of violence. this paper delves into the strategies for detecting fake news and the complexities of distinguishing between the types of fake news. deception, manipulation, and polarization are among the negative impacts that detection seeks to prevent by combating the spread of misleading information across various platforms, including social media and messaging apps. this pressing issue motivates us to dedicate our efforts to this work. the absence of relevant data made it difficult to create a large corpus to identify and categorize false news in malayalam. creating a sufficiently large and diverse dataset is more challenging for malayalam, which is the topic of extensive research due to the absence of malayalam-specific resources. the lack of data makes developing and evaluating reliable identification algorithms more challenging and highlights the urgent need for coordinated efforts to offer datasets related to malayalam. in addition, we explore various machine learning and deep learning algorithms to develop methods to detect fake information effectively. the morphological complexity and the complicated structures of malayalam words and sentences made the feature extraction and classification model development a challenging task, needing the use of advanced techniques in order to capture the complex aspects of the language effectively. the state-of-the-art multilingual bert models were utilized to transform the input sequence into embeddings, whereas support vector machine (svm), random forest (rf), logistic regression (lr), decision tree (dt) and k-nearest neighbour (knn) classification algorithms were employed for categorizing a fake content into different classes. according to the study by @xcite , the dravidian languages, including tamil, telugu, malayalam, and kannada, have unique characteristics. the words are formed by adding affixes to the root word, making it challenging to find the meaning of the words without knowing the con-text in which they are used. the paper leverages the strengths of dilated temporal convolutional networks (dtcn) and integrates them with bidirectional lstm (bilstm) and a contextualized attention mechanism (cam). the dtcn is employed to capture temporal dependencies, bilstm to seize long-range dependencies, and cam to emphasize important information from the news while neglecting irrelevant content. the authors applied an adaptive cyclical learning rate with an early stopping mechanism to improve the model's convergence rate and accuracy. the dataset consists of news articles represented by (d 1 , y 1 ), (d 2 , y 2 ), ..., (d n , y n ), where d represents the article and y its corresponding label. the researchers constructed a model to predict the label for each news article. a bala and p krishnamurthy employed the muril model in @xcite to detect fake news. muril was refined by supervised learning on a handpicked dataset of labelled posts, comments, and keywords in dravidian languages. the process of fine-tuning allowed the algorithm to distinguish between true and fake news. the muril model examines textual information in each news to anticipate the classification and extracts semantic features. with the help of a sizable corpus of data from several indian languages, muril is a transformer-based architecture that has been pre-trained to capture linguistic subtleties and semantic correlations unique to the languages in the dataset. @xcite proposed that data preprocessing is important for fake news detection since the appropriate form of data is required for training ml models. first data cleaning is done to eliminate punctuations, special characters and other html elements that don't add anything to the meaning of the news. the text is then separated into distinct words to produce a structured representation. words are shortened to their base form using stemming processes to maintain consistency. finally the text is vectorized before feeding it to the machine learning model. various models like the bert, albert, xlnet, m-bert are used in this paper and m-bert comes out on top with an accuracy of 0.74. m-bert is a version of bert which supports multilingual text feasibly. it is trained on a combination of monolingual and multilingual data by which it gains the ability to produce language representations of languages from different origins. fine-tuning the model on task-specific labelled data across many languages is a necessary step. @xcite deployed fake news detection models for malayalam. in this work, the punctuations and special characters were removed in data pre-processing and the text is converted into its equivalent english form which is useful for classification. an ensemble machine learning classifier was proposed in this paper to identify fake news. @xcite developed a xlm-r model for fake detection in malayalam. the xlm-roberta model is also a multilingual variant of the bert based transformer model. it consist of self-attention mechanisms which enables it to learn contextualized word embeddings which helps in capturing relationships between words in a sentence, ultimately tuning the model to encode the semantic information of the input text effectively. the model is fine-tuned using an annotated malayalam fake news dataset. it allows the model to learn specific patterns and linguistic characteristics of fake news in malayalam. the news is labelled genuine or fake by augmenting the model with a classification layer on the top. the parameters of the model are updated during the fine-tuning process. bayesian optimizer wass used to find the optimal hyperparameters for the deep learning based model which maximizes the model's performance. the proposed xml-roberta model achieved a f1-score of 87%. according to @xcite , the fake news detection problem is often viewed as a classification problem rather than a regression problem since regression gives us an output of a numeric score of the integrity of the data. pre-processing steps followed in this work includes tokenization, stemming and weighting of words. the input texts were converted into features using tf-idf. though various machine learning models were used for fake news detection, the neural network based model achieved the highest accuracy in detection. attention mechanisms are incorporated into neural networks to boost their performance and accuracy. @xcite implemented three different variations of neural networks. the first model utilizes tf-idf with dense neural networks. this model takes the tf-idf vector of the headline pair's cosine similarity, a standard practice to measure similarity between non-zero vectors, as input and predicts the output. the vectors are passed to the dense network layers, and the final dense layer predicts the output label for the text news. the second model employs a bag of words vector with deep neural networks (dnn). it uses a simplified vector space embeddings to represent text. the third model incorporated a pre-trained word embedding model trained using neural networks. for this neural network architecture, word2vec was employed to represent words in a 300-dimensional vector space, and these embeddings are fed into the classification model. among these, the tf-idf-based model was the best-performing model. to address the potential overfitting of the neural network model, various regularization techniques such as l2 and early stopping, are deployed to improve generalization. this model has demonstrated superior performance compared to existing model architectures, achieving an accuracy of 94% on the test data. ## fake news dataset in malayalam even though there are datasets available for checking whether news is fake, there are no datasets available in malayalam to check how much misinformation a news carries, which motivated our research. therefore, we refer to various fact-checking websites to prepare an authentic dataset to measure different levels of misinformation in the news. this process posed different challenges. we addressed this issue by selecting the factchecking pages of the mainstream media in malayalam. the list of websites from where the data was collected are listed below the collected dataset was divided into train data and test data. the class-wise statistics of the train and test datasets are given in ## methodology the block diagram of the proposed model is shown in figure 1 and the steps involved in the proposed methodology are described below. the classification of fake news into distinct subcategories was modelled as a text classification problem in which a sequence of tokens served as the input, and a class label representing a fake news category was considered the output. we employed bert-based multilingual models to generate embeddings for the input token sequence. besides, machine learning classifiers were utilised to determine the function that converts the embedding into the appropriate labels. the collected data was divided into training and test sets. both training and test datasets were converted into vector representation using bert-based models. we used multilingual sentence transformers for transforming input text sequences into embeddings. the resultant features and their labels were used for building the classifier. since the number of data points is less in the training data, we decided to implement a cross-validation-based grid search approach to fix the optimal hyperparameters of each classification model. the best estimators were used to train the model using the training data. ## experiments this section provides an overview of the five experiments conducted. we considered four multilingual bert models and one malayalam-specific bert model for generating the embeddings. the input sequences were fed into the sentence transformer designed using the abovementioned bert models to generate the feature vector. the multilingual bert models considered for this work are -bert multilingual-cased, muril, labse, and indicbert. the classification of news into different categories was modelled using five different classification algorithms -support vector machine (svm), random forest (rf), decision tree (dt), logistic regression (lr) and k-nearest neighbor (knn) @xcite . the following subsections explain the performance of different classifiers with each embedding model. the first experiment uses the malayalam bert model @xcite . this model is trained on publicly available malayalam monolingual datasets. the performance of each classifier with the embeddings generated using the malayalam bert model is given in in the second experiment, we used the multilingual bert base-cased model @xcite to compute the vector representation for the input data. this model is pre-trained on a large corpus of multilingual data in a self-supervised fashion. it is pre-trained with data collected from 104 languages. the performance of each classifier is shown in the best-performing classifiers in terms of accuracy are svm and rf. both models achieved an accuracy of 74.10%. nevertheless, lr demonstrated the best performance with multilingual bert-based features regarding the f1 score with an f1 score of 0.3353. the optimal hyperparameters for developing the lr model were @xmath0.1, @xmath1, whereas svm and rf were built using the parameters @xmath2, @xmath3.1, @xmath4, respectively. the third experiment used the muril @xcite model for generating the embedding. this model is pre-trained using 17 indian languages. this model is pre-trained on translation and their transliterated counterparts. in the fourth experiment, the labse model was used to transform the input text into embeddings. this model is trained and optimized for bilingual sentence pairs. the performance scores of different classifiers used in this experiment are shown in in the fifth experiment, we utilized indicbert @xcite , a multilingual model pretrained on 12 major indian languages. the result of this experiment is shown in among all the classifiers, lr achieved the highest f1 score with four feature embeddings, and svm demonstrated the best performance with malayalam bert, with a score of 0.3392. the best score attained by lr was 0.3393 labse embeddings. lr exhibited comparable performance with an f1 score of 0.3052 with malayalam bert embeddings. ## conclusion this paper proposes a new dataset to categorize fake news in malayalam into different fake categories based on the degree of falseness. to the best of our knowledge, this is the first dataset curated for fake news classification in malayalam. in addition, we developed baseline models for identifying the fake category of false news in this work using various multilingual bert models and machine learning classifiers. among all the models, the logistic regression model trained over labse features was the best, with an f1 score of 0.3393. the high imbalance in the training data significantly affected the model's performance. the dataset exhibits a significant imbalance, potentially resulting in model biases favouring the majority class. potential solutions to this problem include implementing cost-sensitive learning and oversampling techniques; these represent the future trajectories of this research.