Hoax Terminators@ LT - EDI 2025: C har BERT ’s dominance over LLM Models in the Detection of Racial Hoaxes in Code-Mixed H indi- E nglish Social Media Data (2025)

Abstract: This paper presents our system for the LT-EDI 2025 Shared Task on Racial Hoax Detection, addressing the critical challenge of identifying racially charged misinformation in code-mixed Hindi-English (Hinglish) social media—a low-resource, linguistically complex domain with real-world impact. We adopt a two-pronged strategy, independently fine-tuning a transformer-based model and a large language model. CharBERT was optimized using Optuna, while XLM-RoBERTa and DistilBERT were fine-tuned for the classification task. FLAN-T5-base was fine-tuned with SMOTE-based oversampling, semantic-preserving back translation, and prompt engineering, whereas LLaMA was used solely for inference. Our preprocessing included Hinglish-specific normalization, noise reduction, sentiment-aware corrections and a custom weighted loss to emphasize the minority Hoax class. Despite using FLAN-T5-base due to resource limits, our models performed well. CharBERT achieved a macro F1 of 0.70 and FLAN-T5 followed at 0.69, both outperforming baselines like DistilBERT and LLaMA-3.2-1B. Our submission ranked 4th of 11 teams, underscoring the promise of our approach for scalable misinformation detection in code-switched contexts. Future work will explore larger LLMs, adversarial training and context-aware decoding.

## introduction racial hoaxes are harmful lies that falsely tie people or groups to crimes or events, often picking on specific ethnic or social communities to spark division or fear. the rise of hateful, racially charged speech-especially on platforms like twitter and facebook where users often blend languages like hinglish (a mix of hindi and english)-poses a serious challenge. the hoaxmixplus dataset, consisting of 5,105 youtube comments, serves as a key benchmark for detecting such harmful content. detecting racial hoaxes on social networks is challenging due to the difficulty in distinguishing truth from falsehood, the sheer volume of posts, and the intentional use of humor by users (santoso et al., 2017). traditional systems often misclassify content due to idioms, slang and subtle contextual cues. to address this, we leverage advanced models like the transformer-based char-bert and llm-based flan-t5 foot_0 , fine-tuned with task-specific instructions, rubrics, and prompt formulations. charbert's character-level embeddings and flan-t5's contextual understanding make them well-suited for interpreting nuanced, deceptive content. this paper presents our submission to the racial hoax detection shared task-a robust system leveraging transformers and llms, structured around three contributions: our approach demonstrates promising results in detecting racial hoaxes on code-switched social media platforms, highlighting the effectiveness of utilizing advanced transformer and llm models alongside innovative data augmentation techniques like back translation, smote and oversampling to tackle class imbalance and improve model generalization. for more details and to access the codebase, visit our project repository: github repository foot_1 . in recent years, the detection of racial stereotypes and hoaxes in social media has become a critical focus of research. (bosco et al., 2023) introduced a method for detecting racial stereotypes in italian social media, focusing on the intersection of psychology and natural language processing (nlp). building on these studies, (schmeisser-nieto et al., 2024) presented stereohoax, a multilingual corpus annotated for racial hoaxes and stereotypes. their work addresses a significant gap in understanding social media reactions to racial hoaxes and offers a valuable resource for future research. (raza et al., 2024) explored the effectiveness of bert-like models and large language models (llms) in the detection of fake news, focusing on the challenges posed by generative ai-annotated data. their comparative evaluation provides a useful perspective on how different models perform in fake news and racial hoax detection tasks. (banerjee et al., 2021) investigated transformer-based models for identifying hate speech and offensive content in both english and indo-aryan languages. their work highlights the effectiveness of transformer models in multilingual environments, particularly in identifying harmful content in social media posts. (guo et al., 2024) conducted a large-scale study on the use of llms for hate speech detection, focusing on the role of prompt engineering in improving the models' contextual understanding. their findings suggest that llms can surpass traditional machine learning models in detecting hate speech when properly prompted. recent work by (carpenter et al., 2024) shows the effectiveness of fine-tuned flan-t5 models in educational settings, supporting our choice of flan-t5 for detecting racial hoaxes in code-mixed hinglish. to address the issue of class imbalance, which is particularly critical in hoax detection where hoax instances are typically underrepresented, we draw inspiration from the dynamically weighted balanced (dwb) loss function proposed by (fernando and tsokos, 2021), which adaptively adjusts loss contributions based on class frequency and prediction confidence, enabling the model to focus on harder minority-class samples and improving generalization in imbalanced settings. additionally, (chakravarthi et al., 2025) presented an overview of the shared task on detecting racial hoaxes in code-mixed hindi-english social media data, further advancing the understanding of racial hoaxes in multilingual contexts. ## dataset description the dataset (?) used in this shared task targets the challenge of racial hoax detection in hinglish social media posts. it comprises real-world, usergenerated content labeled with binary annotations: a label of 1 signifies the presence of a racial hoax, while 0 indicates a non-hoax instance. however, the dataset is notably imbalanced, with a significantly higher number of non-hoax examples. as illustrated in the pie charts of figure 1 , the training set contains 2,319 non-hoax cases versus only 741 hoax cases. the validation and test sets each contain 774 non-hoax and 247 hoax samples, creating an imbalance that can trip up standard classification models. these models often lean toward the majority class, making it tough to properly learn from the smaller hoax class. to tackle this, we applied methods like oversampling, smote and loss function adjustments to better emphasize the minority class and enhance the model's effective-ness. ## system in this section, we describe the methodologies employed for detecting racial hoaxes in social media content using two distinct approaches: transformer models and large language models (llms). ## experiments and results in this section, we are presenting the experiments and results of our approaches-transformer-based and large language models for racial hoax detection in hinglish social media text. ## error analysis in the error analysis, we evaluated the performance of the charbert model using the confusion matrix for racial hoax detection as shown in figure 4 . the charbert model correctly identified 677 non-hoax instances but misclassified 121 hoaxes as non-hoaxes and 126 non-hoaxes as hoaxes. these errors emphasize the need to reduce false negatives for better racial hoax detection. the confusion matrix in figure 5 shows the confusion matrix with 676 true negatives, 124 false positives, 98 false negatives, and 123 true positives. these results highlight that the model performs well on non-hoax instances, with a solid precision of 87%, but struggles with hoax classification, showing a lower precision of only 50%. this imbalance can be attributed to the class distribution, as hoax examples are underrepresented in the dataset. the model tends to misclassify sarcastic posts as non-hoax (false positives) and hoaxes with neutral phrasing as non-hoax (false negatives). augmentation strategies, such as back translation and smote, helped mitigate some of these errors, but the challenge of distinguishing between subtle context variations remains. additionally, short and noisy comments presented difficulties due to their inherent context ambiguity, further complicating accurate classification. in addition to the issues identified with sarcasm and neutral phrasing, the charbert model also faced difficulties when dealing with informal language and slang, common in hinglish social media posts. this led to occasional misclassifications, especially when the context was subtle or ambiguous. despite these challenges, the use of data augmentation techniques, like back translation and smote, improved the model's performance by generating more diverse training examples. however, further improvements in handling noisy and short-text comments, along with enhancing the model's ability to detect nuanced hoaxes, will be necessary to address these shortcomings. ## conclusion we introduced an innovative fine-tuned system for racial hoax detection in code-mixed hindi-english youtube comments. specifically, we fine-tuned the charbert and flan-t5 models on the hoaxmixplus dataset, leveraging data augmentation techniques (including back translation and smote), weighted loss optimization, and hinglish-specific preprocessing to address challenges such as class imbalance, linguistic diversity, and contextual ambiguity. our best-performing model, based on flan-t5, achieved a macro f1 score of 0.69 on the test set, while the charbertbased model reached 0.70. these results underscore the effectiveness of integrating augmentation techniques with transformer-based architectures in low-resource, code-mixed settings. future work will explore larger language models (llms), context-aware decoding tailored to hinglish nuances, and constraint-based structured generation to further improve hoax specificity. additionally, we plan to extend this research to other multilingual social media platforms and explore real-time detection mechanisms for dynamic hoax identification. another promising direction involves finetuning models on even more diverse datasets to improve their generalization and robustness. overall, this work demonstrates the potential of advanced nlp models in combating harmful misinformation in underrepresented languages and settings. moreover, the findings highlight the importance of continued innovation in model architecture and training techniques to address the evolving nature of misinformation across diverse linguistic landscapes.