D ravidian Fake News Detection with Gradient Accumulation based Transformer Model (2023)

Abstract: The proliferation of fake news poses a significant challenge in the digital era. Detecting false information, especially in non-English languages, is crucial to combating misinformation effectively. In this research, we introduce a novel approach for Dravidian fake news detection by harnessing the capabilities of the MuRIL transformer model, further enhanced by gradient accumulation techniques. Our study focuses on the Dravidian languages, a diverse group of languages spoken in South India, which are often underserved in natural language processing research. We optimize memory usage, stabilize training, and improve the model’s overall performance by accumulating gradients over multiple batches. The proposed model exhibits promising results in terms of both accuracy and efficiency. Our findings underline the significance of adapting state-ofthe-art techniques, such as MuRIL-based models and gradient accumulation, to non-English languages to address the pressing issue of fake news.

## introduction in the digital age, the rapid proliferation of fake news has emerged as a formidable challenge, permeating every facet of society and eroding trust in information sources @xcite . addressing this issue is paramount, and the ability to detect false information, particularly in non-english languages, represents a critical frontier in the combat against misinformation. this research endeavors to present a novel approach to tackle the problem of fake news detection within the linguistic landscape of dravidian languages. dravidian languages, a diverse and linguistically rich group primarily spoken in south india, often find themselves underserved in the realm of natural language processing research @xcite . the unique characteristics and linguistic nuances of these languages necessitate dedicated attention. our study aims to bridge this gap by introducing a pioneering solution that harnesses the capabilities of the multilingual representations for indian languages (muril) @xcite transformer model, a state-of-the-art language representation model designed for multilingual tasks. furthermore, we enhance the muril transformer model's performance by incorporating the gradient accumulation (lamy-poirier, 2021) technique. this approach offers multiple advantages, such as optimized memory usage, stabilized training processes, and improvements in overall model efficiency. by accumulating gradients over various batches, we mitigate memory constraints while maintaining high accuracy. additionally, we incorporated adaptive learning rate scheduling, a dynamic technique that adjusts the learning rate during training, to ensure our model trains stably and efficiently, eventually improving its ability to converge to a better solution. our findings highlight the substantial impact of adapting cutting-edge techniques to non-english languages, specifically dravidian languages @xcite , in the acute and crucial quest to combat the dissemination of fake news. additionally, we compared the proposed model with other three multilingual models, like multilingualbert (mbert) @xcite , cross-lingual language model for robustly optimized bert (xlm-roberta) @xcite , and indicbert @xcite . this study not only showcases the potential of our approach but also contributes valuable insights to the broader field of natural language processing and misinformation detection, offering a robust foundation for developing tools and strategies to combat misinformation in linguistically diverse landscapes. the main contributions of this research are: the prevalence of fake news, misinformation, and disinformation in the digital age has attracted substantial research attention in recent years. while the majority of existing studies concentrate on fake news detection and classification in widely spoken languages, there needs to be more research focusing on regional languages, such as those within the dravidian language family. this section provides a comprehensive overview of the literature on fake news detection, with a specific emphasis on studies pertaining to regional languages. till now, limited research has been conducted on the detection of fake news in low-resource languages, and the challenges associated with this issue have been addressed in @xcite . a bert-based multilingual model is employed @xcite to identify fake news within the context of covid datasets in indic languages, resulting in an f1 score of 89%. @xcite ) introduced an ensemble model for the purpose of fake news detection and explored the challenges associated with cross-domain intractability. their approach involved a combination of models, including the embedding lstm network, depth lstm, linguistic inquiry, and word count in conjunction with a cnn classifier. additionally, n-gram models were utilized in combination with a cnn classifier to extract relevant features from the news data. to optimize the ensemble learning model, the authors employed the self-adaptive harmony search algorithm, allowing them to determine the most suitable weights for the constituent models. @xcite developed a model to identify offensive language in code-mixed youtube comments across three dravidian languages. this modal employs tf-idf character n-grams, leverages pre-trained muril embeddings and utilizes logistic regression and linear svm for classification tasks. they achieved impressive weighted f1 scores of 64%, 95%, and 71% in kannada, malayalam, and tamil, respectively. @xcite employed three methods to detect and classify youtube comments in tamil and tamil-english code-mixed formats: ensemble models, recurrent neural networks, and transformers. their evaluation of these methods in the tamil dataset revealed that muril and xlm-roberta were the top-performing models, achieving a macroaveraged f1 score of 43%. furthermore, muril and m-bert produced outstanding results in the context of code-mixed data, yielding a macroaveraged f1 score of 45%. ## training in this research, we used the pytorch lightning framework @xcite with an 8gb gpu. in comparison to pytorch, the pytorch lightning framework showcased notable improvements in terms of speed and efficiency. this significant advancement can be credited to the implementation of mixed precision within the pytorch lightning framework. mixed precision leverages both 32 and 16-bit floating-point formats, effectively curbing memory consumption during model training. the outcome is a remarkable boost in performance, resulting in a noteworthy up-to-threefold increase in speed on contemporary gpus @xcite . we employed a transformer model with gradient accumulation to enhance the efficiency of our fake news detection system. figure 2 shows the four multilingual transformer models with and without gradient accumulation (ga) technique over the dravidian_fake dataset, and it clearly describes that the multilingual transformer model with gradient accumulation has less gpu memory usage compared to the model without gradient accumulation technique. the hyperparameters used in this research are shown in gradient accumulation is a technique used to address memory constraints when training deep learning models, particularly on gpus with limited memory capacity. setting the gradient accumulation size to 4 means that instead of updating the model's parameters after processing each batch of data, the gradients are accumulated over four consecutive batches before performing a parameter update. this accumulation effectively simulates a larger batch size, which can lead to more stable training and better convergence. it helps in managing gpu memory efficiently during training, making it feasible to train complex models on hardware with limited memory. the batch size refers to the number of data samples that are processed together in one forward and backward pass during training. in this case, a batch size of 8 indicates that eight data samples are processed simultaneously in each iteration. a smaller batch size can reduce memory usage but may lead to noisy gradients and slower convergence, while a larger batch size can provide more stable gradients but require more memory. the combination of a batch size of 8 and a gradient accumulation size of 4 allows the training process to effectively simulate a batch size of 32 (8 * 4), striking a balance between memory efficiency and training stability. we employed the scheduler returned by get_linear_schedule_with_warmup for adaptive learning. this scheduler adjusts the learning rate during training to help improve model convergence. specifically, it increases the learning rate linearly during the warm-up phase and then decreases it linearly during the remaining training steps. the dataset was divided into training and testing sets using an 80-20% split, maintaining the integrity of the data distribution. furthermore, the testing set was further partitioned into a 90-10% split to create a separate validation subset, which was crucial for monitoring the model's performance and preventing overfitting. ## experimental results and discussions in this research, we utilized the muril transformer model with gradient accumulation and adaptive learning. we conducted a comparative analysis of our proposed model against three other transformer models: mbert, xlm-roberta, and in-dicbert. in comparing the models with and without gradient accumulation, the gradient accumulation models exhibited higher accuracy and f1 scores. specifically, when contrasted with mbert, the version with gradient accumulation displayed a 1.5% higher accuracy and a 0.75% higher f1 score. in the case of xlm-roberta, the model with gradient accumulation outperformed its non-gradient accumulation counterpart with a 2% higher accuracy and a 2.5% higher f1 score. furthermore, when compared to indicbert, the indicbert model with gradient accumulation achieved a 1.35% higher accuracy and a 1.23% higher f1 score. finally, relative to muril, the muril model with gradient accumulation exhibited 0.6% higher accuracy and a 1.5% higher f1 score. significantly, indicbert and muril outperformed mbert and xlm-roberta in terms of accuracy and f1 scores, owing to their specialized training in indian languages, including dravidian languages. ## conclusion the research presented in this study addresses the critical issue of fake news detection within the linguistic context of dravidian languages. by developing the dravidian fake news classifier, we have demonstrated the efficacy of leveraging stateof-the-art natural language processing techniques, specifically the muril model with gradient accumulation for reducing gpu memory usage in training the model, to detect fake news in languages such as tamil, telugu, malayalam, and kannada. the experimental results reveal that the dravidian fake news classifier can remarkably differentiate between fake and legitimate news articles in dravidian languages. the proposed model's accuracy of 92.83% and f1 score of 92.62% on average of the four dravidian languages in the test phase attest to its proficiency. the study underscores the importance of language-specific fake news detection by customizing the model for dravidian languages. it recognizes the linguistic nuances present in dravidian languages and the necessity of considering these aspects when addressing misinformation. in the future, we will explore multimodal approaches that combine textual information with other modalities, such as images and videos, for fake news detection in dravidian languages.