byte S ized LLM @ D ravidian L ang T ech 2024: Fake News Detection in D ravidian Languages - Unleashing the Power of Custom Subword Tokenization with S ubword2 V ec and B i LSTM (2024)

Abstract: This paper focuses on detecting fake news in resource-constrained languages, particularly Malayalam. We present a novel framework combining subword tokenization, Sanskrit-transliterated Subword2vec embeddings, and a powerful Bidirectional Long Short-Term Memory (BiLSTM) architecture. Despite using only monolingual Malayalam data, our model excelled in the FakeDetect-Malayalam challenge, ranking 4th. The innovative subword tokenizer achieves a remarkable 200x compression ratio, highlighting its efficiency in minimizing model size without compromising accuracy. Our work facilitates resource-efficient deployment in diverse linguistic landscapes and sparks discussion on the potential of multilingual data augmentation. This research provides a promising avenue for mitigating linguistic challenges in the NLP-driven battle against deceptive content.

## introduction the surge in online social media platforms has transformed communication dynamics, facilitating seamless information exchange, dialogue, and realtime awareness of current events. however, this unprecedented connectivity has also given rise to a worrisome proliferation of misinformation, commonly known as fake news @xcite . in response to this escalating issue, we introduce the fake news detection in dravidian languages challenge-dravidianlangtech@eacl 2024 foot_0 . this initiative addresses the critical need for robust fake news detection mechanisms, particularly in dravidian languages. the challenge focuses on developing effective models capable of distinguishing between authentic and fake content across various social media platforms, such as twitter and facebook. task 2, the fakedetect-malayalam shared task, is a crucial platform for researchers addressing the challenge of identifying fake news within malayalam-language articles. in an era of information overload, precise detection of misinformation is vital for trustworthy communication. the task's primary goal is to motivate participants to develop effective models for accurately classifying fake news into categories like false, half true, mostly false, partly false, and mostly true. this research paper, inspired by the task, explores innovative approaches for fake news detection in dravidian languages, focusing on malayalam, aiming to contribute novel insights and methodologies to advance the state of the art. our study focuses on fake news detection in south indian social media, introducing an innovative approach featuring a custom-designed tokenizer and a bidirectional long short-term memory (bilstm) architecture. a key innovation is our tokenizer, which reduces model sizes, improves efficiency, and addresses challenges related to realtime deployment, enhancing the practicality of deploying detection systems in dynamic online environments. the implementation of bilstm, coupled with our specialized tokenizer, exhibits significant enhancements in fake news detection accuracy, demonstrating heightened sensitivity to linguistic nuances. the use of compact models effectively addresses inference challenges, ensuring swift realtime detection and practical deployment. our work establishes a framework for fake news detection, providing valuable insights into model size, inference speed, and the challenges of real-time deployment in countering online misinformation within dravidian languages. this paper explains our approach, technical improvements, and findings, thoroughly investigating the detection of fake news in dravidian languages. our goal is to contribute not only to addressing the challenges of fake news in malayalam but also to enhancing our understanding of effective detection methods suitable for various linguistic contexts. ## related work the growing concern over disinformation and propaganda has led to increased research on fake news detection. in recent works, @xcite focused on detecting fake news in dravidian languages using transfer learning with adaptive fine-tuning, emphasizing different techniques with transformer-based models. keya et al. (2022) employed a pretrained bert model with data augmentation, comparing their results with twelve different models. goldani et al. (2021) utilized capsule networks with varied architectures and n-gram levels for feature extraction. in the context of languages other than english, studies like gereme et al. (2021) and saghayan et al. (2021) explored fake news detection in amharic and persian, respectively. chu et al. (2021) investigated the ability to generalize fake news detection models across languages, finding the bert model effective. faustini and covões (2020) covered multiple languages, including slavic, latin, and german, emphasizing the need for fake news identification in resource-poor languages like dravidian languages. additionally, vijjali et al. (2020) addressing covid-19 fake news and proposing a two-stage automated pipeline employs bert and albert models for efficient verification. notably, many studies primarily focus on english and other major languages, highlighting the potential for advancements in resource-poor languages. ## methodology this section unveils the intricate details of our innovative architecture, which seamlessly integrates two crucial components: a dynamic subword embeddings module and a robust bilstm tagger module. we embark on a journey through the art of data preprocessing, the finesse of subword tokenization, the mastery of embedding training, and ultimately, the orchestration of our advanced classifier. ## experimental setup our experimental setup aims to showcase the effectiveness of our proposed approach for fake news detection in dravidian languages, with a specific focus on malayalam. utilizing the custom subword tokenizer "voweltoken," which considers vowel boundaries for efficient tokenization, and subword2vec for generating 100-dimensional subword embeddings, we meticulously trained the embeddings on a 1gb monolingual malayalam text corpus. a sanskrit transliterator was employed to aid the training process. despite limiting the training to a single epoch, exceptional results were achieved, with a perplexity of 1.07 on a 10mb development dataset, highlighting the effectiveness of our approach. the final 100-dimensional embedding size after training was 5.92mb, striking a balance between efficiency and accuracy. to assess the effectiveness of the subword embeddings, we seamlessly incorporated them into our bilstm-based model architecture. the clas-sificationmodel comprises a sub-word embedding layer, a bi-directional lstm layer, and a linear classification layer, leveraging subword embeddings obtained from voweltoken. the bilstm layer is configured with an input size of 100 and a hidden size of 128. moreover, the model employs the adam optimizer with a learning rate of 0.001 throughout the training process. our model underwent rigorous training on task 2, the multi-class classifier, to comprehensively assess its performance. this experiment aimed to demonstrate the robustness of our custom subword tokenizer and explore the contribution of subword embeddings within the bilstm-based classification model to fake news detection in the context of dravidian languages, especially malayalam. evaluation metrics, including recall, precision, f1 score, and accuracy, were used to measure the model's effectiveness. ## experimental results and discussions motivated by the challenges of obtaining large multilingual datasets, we intentionally trained our subword embeddings on a limited 1gb monolingual text corpus. despite this constraint, the embeddings exhibited competitive performance, achieving remarkable accuracy on unseen data. the classification report (see the overall accuracy of 0.66 showcases the model's competence in classifying news articles into various categories. it is essential to note that the model achieves a reasonable performance, considering the complexity of the task and the limited size of the model (6.87mb), ensuring efficient deployment with approximately 10x faster inference compared to large language models (llms). it's noteworthy that acquiring training data for embedding training from social media poses a significant challenge. despite these difficulties and limited data embeddings, our model still achieves top-notch performance, as demonstrated by our bytesizedllm team securing the 4th rank overall in the task 2 competition (see the effectiveness of our subword tokenization is evident in low perplexity after just one epoch, showcasing the model's potential. unlocking its full capabilities involves leveraging more extensive training data for heightened accuracy and generalization. while expanding the dataset may increase subword tokens, the model size is expected to remain manageable due to the saturation of subword tokens within the initial 1gb of text. ## conclusion and future work our innovative fake news detection approach for dravidian languages leverages voweltoken, a custom subword tokenizer capturing vowel nuances within dravidian languages. this granular understanding improves subword embedding generation, enhancing model performance. the bilstm architecture coupled with custom subword embeddings demonstrates efficient information extraction and classification, achieving promising results despite limited training data. the model's compactness facilitates implementation, further contributing to real-world applicability. future work will explore larger social media and multilingual datasets and investigate advanced embedding techniques like contextualized embeddings, aiming to further improve the model's performance and unlock its full potential in diverse dravidian language contexts.