ML & AI _ IIITR anchi@ D ravidian L ang T ech:Leveraging Transfer Learning for the discernment of Fake News within the Linguistic Domain of D ravidian Language (2023)

Abstract: The primary focus of this research endeavor lies in detecting and mitigating misinformation within the intricate framework of the Dravidian language. A notable feat was achieved by employing fine-tuning methodologies on the highly acclaimed Indic BERT model, securing a commendable fourth rank in a prestigious competition organized by DravidianLangTech 2023 while attaining a noteworthy macro F1-Score of 0.78. To facilitate this undertaking, a diverse and comprehensive dataset was meticulously gathered from prominent social media platforms, including but not limited to Facebook and Twitter. The overarching objective of this collaborative initiative was to proficiently discern and categorize news articles into either the realm of veracity or deceit through the astute application of advanced machine learning techniques, coupled with the astute exploitation of the distinctive linguistic idiosyncrasies inherent to the Dravidian language.

## introduction text classification has emerged as a critical task that has garnered significant attention and contributions from researchers and practitioners worldwide in recent years. the rapid increase in user enrollment on social media platforms has led to an unprecedented surge in the availability of textual data. this abundance of textual data has played a pivotal role in various domains, including business and research, shaping strategies, and driving advancements in software and applications. companies rely on textual data, particularly user reviews and comments, to formulate their business strategies and make informed decisions about their products and services. by analyzing the sentiment and content of textual data, businesses gain valuable insights into customer opinions, preferences, and trends. furthermore, researchers have recognized the potential of textual data analysis in advancing fields such as natural language processing, sentiment analysis, and information retrieval. the exploration of textual data has paved the way for the development of more intelligent software and applications that can understand and process human language more effectively. the analysis of textual data assumes even greater significance when considering its multilingual nature. with a diverse range of languages spoken worldwide, understanding and analyzing textual data across different languages is crucial. while most contributions in the field of text classification have focused on the english language domain, it is essential to explore the characteristics and challenges posed by other languages. low-resource languages, such as the dravidian languages, often face a scarcity of available methodologies and resources for effective text analysis. this poses a unique challenge that requires innovative approaches and solutions. the impact of textual data goes beyond business and research, particularly in the era of social media and digital communication. a study conducted by mit foot_0 revealed a disconcerting trend: fake news spreads faster than accurate information, particularly on platforms like twitter, where it can propagate at a rate 10 to 20 times faster. this poses a significant challenge to individuals, organizations, and society as a whole, as the proliferation of misinformation can lead to detrimental consequences. trust in news sources and the ability to discern reliable information become paramount in such an environment. addressing this issue becomes even more challenging when dealing with low-resource languages like the dravidian languages. the scarcity of methodologies and resources specifically tailored for these languages poses a barrier to effective fake news detection and mitigation. however, recognizing the urgency and importance of finding solutions, competitions and collaborative initiatives have emerged, bringing together great minds from around the world to tackle this pressing problem. these platforms foster innovation, encourage the development of novel methodologies, and promote advancements in technology that can benefit the human race. with the aim of contributing to this endeavor, we embarked on an experiment to identify fake news in the malayalam language, a dravidian language spoken primarily in the indian state of kerala. leveraging various machine learning and deep learning techniques, we sought to devise a suitable solution that could effectively detect and classify fake news in this language. our experimentation involved training and fine-tuning models using state-of-theart approaches, with a particular focus on transfer learning. by fine-tuning indic-bert, a language model specifically designed for indian languages, we aimed to leverage its pre-trained knowledge to improve the accuracy and effectiveness of our classification system. the results of our experiment were promising, with an achieved macro f1-score of 0.78. this performance placed us in an admirable 4 th position in the competition @xcite , reflecting the efficacy of our approach and the efforts invested in tackling the challenges specific to the malayalam language. by harnessing the power of transfer learning and combining it with domain-specific fine-tuning, we enhanced our system's capabilities and achieved commendable results in detecting and classifying fake news. rest of the paper summarised in different sections. related works are presented in section 2 followed by task description and dataset description in section 3 and section 4 respectively. in section 5, we presented our detailed methodology and validation results followed by section 6, where we presented the results of test data and analysis of the the results. finally, we summarise our work in section 7. ## related work due to their potential impact on public discourse and the spread of disinformation, the identification and mitigation of fake news on social media have become key fields of research. this part includes an extensive assessment of the pertinent literature and covers significant contributions to the study of how to spot false news. in their seminal work, the work @xcite adopt a data mining perspective to address the challenge of detecting fake news on social media. their research underscores the significance of employing data mining techniques to extract features and patterns from social media content, enabling accurate identification of fake news. the authors propose a framework incorporating textual, visual, and social features to discern instances of fake news. by highlighting the role of data mining in tackling the issue of fake news on social media platforms, this study contributes valuable insights to the field. the article (wang, 2017) introduces the benchmark dataset "liar, liar pants on fire," specifically designed for fake news detection. this dataset consists of statements that have been labeled with varying degrees of truthfulness, enabling researchers to develop and evaluate models for fake news detection. the author emphasizes the significance of high-quality datasets in advancing research in this domain and provides a valuable resource for benchmarking and comparing different approaches. the researches @xcite @xcite ) explore deep learning approaches for fake news detection. they extensively review a range of deep learning models and techniques, including convolutional neural networks (cnns), recurrent neural networks (rnns), and attention mechanisms, applied in the context of fake news detection. the survey provides valuable insights into the strengths and limitations of different deep learning approaches and highlights their potential for detecting fake news. conducting a comparative study of computational fake news detection on social media, the research (manish et al., 2022) evaluate the performance of various machine learning and deep learning algorithms. the authors assess different features, such as linguistic, stylistic, and temporal characteristics, and examine their effectiveness in distinguishing between fake and real news. this comparative analysis offers valuable insights into the strengths and limitations of different computational approaches for fake news detection. in conclusion, there are many different viewpoints and methods covered in the literature on detecting fake news on social media. the articles discussed in this part highlight how crucial it is to use data mining methods, benchmark datasets, deep learning models, and comparative analyses to progress the field of false news detection. these studies' findings, benchmark datasets, and suggested frameworks are all useful tools for creating computational techniques that can stop fake news from spreading on social media platforms. we attempted to draw inspiration for our work from these references, and after comparing our dataset with theirs, we attempted to develop some useful approaches. ## task description dravidianlangtech presents an intriguing competition titled "fake news detection in dravidian languages." the primary objective of this task is to develop a proficient classification system that can effectively distinguish between original and fake news in social media text written in dravidian languages. the dataset for this task comprises social media posts extracted from diverse platforms like twitter, facebook, and others. participants have been provided with social media texts in dravidian languages. they are tasked with building machine learning or deep learning models to accurately classify the given content as either fake or original news. the focus of the competition revolves around the ability to discern authentic information from fabricated or misleading content within the context of dravidian languages.the outcomes of this task hold significant implications for enhancing the credibility and trustworthiness of information shared on social media platforms in dravidian linguistic domains. furthermore, developing effective classification models will contribute to empowering users with reliable information, foster informed decision-making, and combat fake news dissemination. ## dataset description the dataset @xcite utilized for the fake news detection competition in dravidian languages comprises a diverse collection of social media posts sourced from prominent platforms, including twitter and facebook. it is segmented into three distinct subsets: a training dataset, a development dataset, and a test dataset. the training dataset furnished to us encompasses a substantial volume of social media posts composed in dravidian languages in code-mixed format. each post within the training dataset is annotated with labels indicating its categorization as either an original or fake news post. the total texts in the training dataset were 3257 in number. these labeled annotations served as the ground truth for training classification models, playing a crucial role in facilitating the development and refinement of machine learning or deep learning models. the development dataset, also referred to as the validation dataset, augments the labeled data provided. it allowed us to evaluate models' performance and fine-tune the hyperparameters during the development phase. the labeled information within the development dataset aids in the assessment of the models' efficacy in accurately classifying social media text as either original or fake news.the total text in development dataset were 815. ## methodology and validation results in our study, we employed a range of methodologies on the development dataset with the objective of identifying the most effective approach for making predictions on the test dataset. different techniques @xcite presents a technique where mbert has been utilized along with word-level language tag to classify the comments. the article @xcite presented an approach where feature extraction has been done in association with transfer learning to perform the classification task. being inspired by the related work, we have tried to introduce methodologies that could prove effective in this task. for example, when working with tf-idf (term frequency-inverse document frequency) features and embeddings-based classification, extensive data pre-processing was undertaken to optimize the performance of the models. however, in the case of fine-tuning indic-bert, this data preprocessing step was not necessary. indic-bert, have been pretrained on a large corpus of social media texts, specifically in indian languages, incorporates the linguistic nuances and contextual information prevalent in social media language, including the presence of emojis foot_1 . consequently, removing emojis in this case could potentially compromise the linguistic semantics captured by indic-bert. data pre-processing was conducted specifically for methods such as tf-idf feature extraction and sentence embedding-based classification. this crucial step encompassed several operations aimed at enhancing the quality and structure of the text data. the initial phase involved the removal of punctuation marks, emojis (only for methodologies of section 5.1 to 5.3) and alphanumeric characters to eliminate noise and ensure a cleaner representation of the textual content. following this, we proceeded with the removal of stop words, which are commonly occurring words that do not contribute significantly to the overall meaning of the text. additionally, we focused on expanding any contracted words to their full forms, allowing for a more comprehensive analysis of the text. the final step of the data pre-processing pipeline involved tokenization, which entailed breaking down the text into individual units such as words or sub-words. this process facilitates subsequent analysis and modeling tasks by providing a structured representation of the text data. furthermore, we applied lemmatization to the tokens, aiming to reduce inflected or variant forms to their base or dictionary form, promoting consistency and coherence within the dataset. by diligently performing these data preprocessing techniques, we aimed to enhance the quality and reliability of the text data, preparing it for subsequent analysis and classification tasks. following the feature extraction stage, we applied several machine learning models and deep learning techniques to perform the classification task. the specific methods utilized are discussed in subsequent subsections of this study, outlining the intricacies and nuances associated with each approach. by exploring and evaluating multiple methodologies, we aimed to identify the most effective techniques for classifying social media texts as original or fake news in the context of our research. ## results and analysis in this section, we present the results of our submitted task. after making predictions using the fine-tuned indic-bert, we were evaluated using the macro f1-score, macro precision, and macro recall. we obtained a macro f1 score of 0.78 on the test dataset. the confusion matrix has been presented in figure 2 that tells the classification of various classes along with misclassified classes. it is a critical tool for analyzing the performance and efficiency of our model. 7 conclusion to summarize, this research study delved into multiple text classification techniques, encompassing bag-of-words based classification, tf-idf-based classification, sentence embedding based classification and the fine-tuning of the pre-trained indic- bert model. each approach exhibited distinct advantages and showcased its effectiveness in accurately categorizing textual data. our model and experiments performed satisfactorily over task; all the results are summarized in the tables. we can enhance the performance by fine-tuning the pretrained model with more data.