Overview of the Shared Task on Detecting Racial Hoaxes in Code-Mixed H indi- E nglish Social Media Data (2025)

Abstract: The widespread use of social media has made it easier for false information to proliferate, particularly racially motivated hoaxes that can encourage violence and hatred. Such content is frequently shared in code-mixed languages in multilingual nations like India, which presents special difficulties for automated detection systems because of the casual language, erratic grammar, and rich cultural background. The shared task on detecting racial hoaxes in code mixed social media data aims to identify the racial hoaxes in Hindi-English data. It is a binary classification task with more than 5,000 labeled instances. A total of 11 teams participated in the task, and the results are evaluated using the macro-F1 score. The team that employed XLM-RoBERTa secured the first position in the task.

## introduction in today's world, social networks play a vital role in how people get their information, but they also make it easier for false claims to spread quickly @xcite . one serious kind of fake story is racial hoaxes, where people wrongly blame a particular person or community for a crime or incident simply because of their race, religion, or background. what makes these hoaxes really dangerous is that they create and spread unfair stereotypes, divide people, and can create real trouble, such as fights or community clashes. the problem becomes even harder to deal with in a diverse country like india, where people often mix languages like hindi and english in their on-line posts, so even tech tools have a hard time spotting these misleading claims. to take a step toward solving this problem, a shared task called "detecting racial hoaxes in code-mixed hindi-english social media data" was launched as part of the lt-edi workshop 2025. as part of this task, a new dataset called hoaxmixplus was introduced. it includes 5,105 youtube comments written in a mix of hindi and english, each carefully marked to show whether it contains a racial hoax. this data set reflects how complex and sensitive language can be when people of different backgrounds talk online. unlike general fake news detection @xcite @xcite , this task focuses on more subtle signs such as blaming someone without proof, hinting at accusations without saying them outright, or using words that carry hidden meaning. spotting these patterns requires systems that can detect deeper language cues and the context behind the words. the primary goal of the task is to benchmark and encourage the development of robust classification models capable of identifying racial hoaxes in a low-resource and highly informal setting. the availability of hoaxmixplus serves not only as a starting point for building such models but also as a valuable contribution to the field of computational social science, particularly in understanding and mitigating the spread of race-based misinformation in south asian digital ecosystems. participants were expected to take a step toward tackling this issue related to code switching, informality, implicit bias, and contextual ambiguity. ## related work in recent years, the rise of social networks has led to an explosion in code-mixed communication, especially in regions that are linguistically diverse like india. hindi, an official and widely spoken language in north india @xcite , is traditionally written in devanagari script. however, informal online interactions have adopted a codemixed setup in which users express their personal thoughts and emotions. these pose significant challenges for automated detection of abusive language and hate speech. traditional corpora and models, trained on monolingual and structured data, often fall short when faced with the informal grammar, intentional misspellings, and hybrid syntax typical of code-mixed texts @xcite @xcite . detecting nuanced social phenomena like hate speech, especially when it is unstructured code-mixed language, more than just large dataset it requires deeply annotated corpora, careful curation, and time-intensive fact-checking processes. although some progress has been made in this direction-for example, italian social media texts @xcite , multilingual datasets in italian, spanish, and french annotated in racial hoax for immigrant stereotype @xcite , english dataset named stereoset @xcite are some of the novel corpus. however, access to similarly robust and diverse datasets for indian code-switched scenarios remains limited. as a result, researchers working in this area face considerable challenges not only in developing accurate detection models, but also in constructing the foundational datasets needed to reflect the complex socio-linguistic realities of code-mixed data and bias online. the development of hindi-english code-mixed datasets for abusive language and hate speech detection presents significant challenges, as outlined in prior studies @xcite @xcite . unlike general hate speech, racial hoaxes demand fine-grained analysis of implicit and explicit biases targeting specific social groups. transformer-based models have been applied in related contexts, such as bangla-english sentiment analysis on youtube @xcite , and swahili-english political misinformation @xcite . approaches such as bilstm-crf architectures @xcite and transformer-based models like hing-bert, hingroberta and hinggpt-pre-trained @xcite have shown promise for pos tagging and sentiment classification in codemixed data. novel language augmentation strategies, including word-level interleaving and latinscript resource integration @xcite , further enhance classification performance combined with dedicated lexicon for latin-script hindi-english words, combined with fine-tuned multilingual bert and muril models, effectively tackles the unique challenges presented by hindi-english code-mixed datasets. these innovations collectively highlight the evolving methodologies and underscore the importance of tailored pre-training and annotation in handling the complexity of hindi-english code-mixed text. ## task description the racial hoax detection task, featured at lt-edi@ldk 2025 foot_0 , challenges participants to develop automated systems capable of identifying racial hoaxes within code-mixed hindi-english social media content. racial hoaxes refer to false statements that use misleading information to falsely accuse people or groups because of their social, ethnic, or religious backgrounds, including caste, nationality, or religion. the task involves classifying user-generated comments, primarily sourced from youtube, into two categories: those containing racial hoaxes (positive) and nonracial hoaxes (negative). the binary classification process becomes challenging because the data contains code-switching, informal language, and sociopolitical complexities. participants are provided with a labeled dataset and are expected to submit predictions on a held-out test set. the evaluation metric for this task is the macro-averaged f1 score, which balances performance across both classes. the research goal of this task focuses on developing multilingual and low-resource natural language processing while prioritizing ethical ai applications for misinformation detection. ## dataset descriptions the hoaxmixplus dataset consists of 5,105 youtube comments that are code-mixed hindi-english to help detect and classify racial hoaxes in social media discourse. racial hoaxes refer to false or fabricated claims that target people or groups based on their social or ethnic identity, such as caste, religion, or nationality. the dataset was created by scraping over 210,000 youtube comments from socio-politically relevant videos using targeted keywords such as dalit, caa-nrc, manipur, rohingyas, khalistan, and kerala story. we selected these keywords to ensure a diverse collection of comments that accurately reflected realworld discourse on potentially sensitive topics. the polyglot language detection library performed two functions: it eliminated code-mixed hindi-english comments, and it eliminated all non-hindi-english code-mixed data and monolingual content. the comments received binary classification labels of positive (racial hoax) for false accusations and fabricated stories, and stereotypes against social group,s and negative (non-racial hoax) for neutral, factual, or non-misleading content. native hindi-speaking annotators performed the annotations while using a custom gpt-4 chatbot for consistency and guidance. the team resolved ambiguous cases by reaching majority consensus, and krippendorff's alpha (α = 0.747) measured the inter-annotator agreement at substantial levels. the dataset contains different code-switching patterns, which include intra-sentential switching (within a sentence), inter-sentential switching (between sentences), and tag switching (inserting single words or short phrases from another language), with most comments written in latin script using hindi grammar and english vocabulary or vice versa. ## methodology the racial hoax detection task participants employed different methods to solve the problem while working toward precisely identifying racially motivated misinformation in noisy hindi-english social media comments. the task required systems to process code-switching linguistic complexity together with the delicate sociopolitical elements that frequently appear in racial hoaxes. the systems needed to maintain strong natural language understanding capabilities while being sensitive to cultural details, which standard sentiment and hate speech detection pipelines typically miss. the binary classification framework of the task required models to achieve both high precision and reliability because false positives and false negatives had significant real-world consequences. the majority of teams began their architecture with advanced multilingual transformer models. the team selected xlm-roberta, muril, mbert, and deberta models because these models showed excellent cross-lingual performance and worked well in low-resource conditions. the preprocessing pipelines adapted the input data through techniques that included devanagari script transliteration and en, hi, and oov language tagging and stopword handling, and social media noise removal of emojis, urls, and usernames. the systems either concentrated on syntactic cleaning or used semantic features and handcrafted lexicons to detect hoax-related patterns. the team applied data augmentation techniques, which included synonym replacement, back-translation, and adversarial examples, to enhance model generalizability and reduce the imbalance between hoax and non-hoax examples. the distinguishing factor between submissions was their effective combination of linguistic heuristics with ensemble strategies and domainspecific knowledge bases. the cuet's_white_walkers (rahman et al., 2025b) team established their system using xlm-roberta as one of its key contributions. the team started by freezing the lower transformer layers during the initial training period to maintain general language representations before fine-tuning the @xcite team concentrated their strategy on muril which represents a transformer model trained on indian languages. the preprocessing began with an intense method to clean and normalize the code-mixed text. the model received transliteration and language tagging to standardize the inputs before undergoing stratified class balancing training. the model concentrated on preserving stability against social media noise and unclear sentence structures. the team em-26 @xcite employed mbert's multilingual features in their approach. the team employed traditional finetuning together with lexicon-aware normalization in their approach. the team focused on hindi words with inconsistent spellings and transliterations through normalization techniques and token filtering to prepare training inputs. the team studied multiple approaches to control model complexity through dropout and attention masking techniques. cvf_nitt implemented an ensemblebased architecture through weighted softmax aggregation of xlm-r, mbert, and muril outputs. the ensemble received advantages from a strong preprocessing pipeline, which eliminated emojis and expanded contractions and standardized slang. they also implemented a language-modelagnostic token weighting scheme to amplify social cues linked with misinformation. kcrl introduced a multi-model ensemble that merged predictions from xlm-roberta, muril, and deberta. their work emphasized cross-verification through handcrafted stereotype dictionaries and explored variations in token-level input formatting. they also curated a preprocessing module to identify and standardize acronyms, which are often misleading or misused in racial hoax discourse. the team hoax terminators @xcite submitted three variations of their system. their first run combined muril and xlm-r in a straightforward ensemble using languagetagged inputs and transliterated text. the second run used a deberta-based model, which incorporated a racial hoax lexicon to introduce inductive bias during learning. the third run used mbert with strong data augmentation techniques such as synonym injection and entity replacement to simulate social media noise and test robustness. the team dll5143a @xcite examined three runs through the bacohoax framework. the first configuration used a muril-based backbone trained on clean, length-filtered samples from hoaxmixplus. the second version incorporated interleaved language tags at the word level and leveraged linguistic cues via attention masking. their final submission utilized deberta, coupled with character-level embeddings and a stereotype term dictionary that informed contextual weightings during classification. altogether, these varied methodologies illustrate how teams tailored their systems not only to meet the task's technical challenges but also to address its social and linguistic intricacies. the strong use of domain knowledge whether in the form of handcrafted features, curated lexicons, or transliterationinformed pipelines proved essential in bridging the gap between model generalization and task-specific sensitivity. these contributions underscore a grow-ing maturity in how the nlp community is approaching misinformation detection, especially in multilingual, code-mixed, and ethically charged contexts. ## result and discussion there were 11 participating teams who applied various approaches to tackle racial hoax detection in noisy hindi-english code-mixed social media comments. ## conclusion we presented the first ever shared task on detecting racial hoaxes in code-mixed hindi-english social media data. we expect this task to have a great influence on low resource languages especially in the nlp domain because we received a variety of submissions with various methodologies. the most successful submissions applied were model finetuning with good training strategies, ensemble based methods and addressing lexicon aware normaliza-tion for code mixed hindi-english words. the prediction evaluation was evaluated with a macro f1 score. the task showcased improvement in training techniques leveraging domain knowledge through tailored features and informed processing approaches has been key to enhancing model performance on specialized tasks reducing the gap between model generalization and task specific language sensitivity.