Word-Level Detection of Code-Mixed Hate Speech with Multilingual Domain Transfer (2025)

Abstract: The exponential growth of offensive language on social media tends to fuel online harassment and challenges detection mechanisms. Hate speech detection is commonly treated as a monolingual or multilingual sentence-level classification task. However, profane language tends to contain code-mixing, a combination of more than one language, which requires a more nuanced detection approach than binary classification. A general lack of available code-mixed datasets aggravates the problem. To address this issue, we propose five word-level annotated hate speech datasets, EN and DE from social networks, one subset of the DE-EN Offensive Content Detection Code-Switched Dataset, one DE-EN code-mixed German rap lyrics held-out test set, and a cross-domain held-out test set. We investigate the capacity of fine-tuned German-only, German-English bilingual, and German-English code-mixed token classification XLM-R models to generalize to code-mixed hate speech in German rap lyrics in zero-shot domain transfer as well as across different domains. The results show that bilingual fine-tuning facilitates not only the detection of code-mixed hate speech, but also neologisms, addressing the inherent dynamics of profane language use.

## introduction warning: this paper contains content that may be offensive or upsetting. ever-growing offensive online contents can negatively impact humans @xcite , which can be addressed with effective detection mechanisms. considerable attention from research to this task has emphasized the need for robust and effective as well as multilingual methods @xcite . one growing challenge in language detection tasks is code-mixing @xcite input and output german hurensohn manchmal denk ich mir einfach nur fick dich @xcite (son of a bitch, sometimes i just think fuck you) english holy shit man , i got head last night @xcite n'] code-mixed ah , was ist das für ein muschibattle @xcite (ah, what a pussy battle is this) code-mixed all the way up von der mittelschicht ['n', 'n', 'n', 'n', 'n', 'n', 'n'] (all the way up from middle class) to address this limitation, we contribute five word-level, manually annotated hate speech datasets, a lexicon-based english and german dataset from social media, a subset of the de-en offensive content detection code-switched dataset (ocd) @xcite , a crossdomain held-out test set and a code-mixed german-english rap lyrics dataset from the german rap domain. german rap, a subgenre of hip hop, is primarily performed in german and is heavily influenced by u.s. american hip hop. it often includes multilingual code-mixing and profane language, with some artists using more than 10 different languages in their lyrics @xcite . as an additional contribution, we evaluate the impact of language-specific fine-tuning on zeroshot domain transfer to code-mixed hate speech as well as across domains, which is compared to the zero-shot performance of state-of-the-art large language models (llms). although the size of the proposed datasets is comparatively limited, this paper contributes a real-world benchmark on multilingual as well as code-mixed word-level hate speech detection, as exemplified in ## related work hate speech detection has been a highly active field of research over the past decades. since, to the best of our knowledge, no previous publication considers code-mixing and word-level annotation, this section focuses on approaches that address code-mixed (profane) language and word-level hate speech detection. ## dataset all five datasets detailed in ## method this section first explains the selection of the pretrained language model and details its fine-tuning procedure to obtain different versions of the model, testing their ability to generalize to code-mixed data and other domains, including zero-shot domain transfer. we also performed comparative zero-shot experiments with llms. furthermore, it details the final hyperparameter settings after optimization. the experiments and their objectives are presented as well as the final evaluation with standard metrics in hate speech detection. ## result in order to determine whether the inclusion of languages involved in the code-mixed hate speech positively impacts the models' performance on zero-shot domain transfer, we first compare the fine-tuning results of the individual models, and then their performance on two held-out test sets on different domains. the three models presented in section 4.1 were fine-tuned and tested on their hate speech in-domain test datasets with the optimized hyperparameter settings, where the german-only dft model achieved an f1 score of 82.11%, the german-english deft model achieved a 91.53% f1 score, and the code-mixed german-english cmm model obtained 88.89%. these results indicate the models' considerably robust performance in their respective domains. the achieved test set f1 score by the cmm is considerably higher than the xlm-r-based 58% reported by @xcite , even though the results are not directly comparable since only a subset of the ocd dataset was used and annotated with word-level hate speech labels. this could be indicative of a better performance of token than sentence classification and labeling, even though this assumption requires further evidence with a direct comparison of these two types of task. zero-shot and cross-domain transfer. the results in the second held-out cross-domain test dataset, only called held-out in baseline zero-shot experiments with llms show that the deft model even outperforms the best performing mistral model and recall tends to be an issue for the tested llms. for gpt-4o the main challenge were multi-word sequences, where a substantial number of words that represent no explicit hate speech were annotated as 'b' after an explicit swear word marked with 'b-b', e.g. motherfucker (b-b) wollen groß verdienen (motherfucker want to earn big) was labeled as @xcite . in fact, multi-word hate speech sequences were an issue for all three llms, since mistral and deepseek assign hardly any 'b' labels. however, mistral correctly identified more bad words than the other two models on both datasets, but deepseek obtained the highest precision possible on the rap dataset. the results indicate that the single domain rap dataset was labeled more successfully by llms than the cross-domain held-out dataset. all xlm-r-based models across both datasets show higher recall than precision results, indicating that they perform better on identifying hate speech sequences than accurately differentiating them from other sequences. this difference between the two metrics is lower for the cross-domain held-out dataset than for the rap dataset, which contains more creative uses of code-mixed hate speech and neologisms. the distribution of gold standard categories for deduplicated swear words in the rap dataset compared to missed swear words in the models' predictions is depicted in figure 1 . it shows that ethnicity, gender, and sex are the most challenging categories for the cmm model, with both ethnic slurs not accurately labeled, 8 out of 19 gender-specific swear words missed, and 10 out of 24 sex-related hate speech sequences incorrectly labeled. sex and gender seem to be the most challenging categories for all models on this comparatively small dataset. for instance, blowt in the sex category as grammatical mixture of the english blow and german blast as third person verb is inaccurately labeled by the dft and cmm model, but correctly detected by the deft model. from the llms, only gpt-4o correctly detects blowt. even etymologically more obscure swear words, such as punanis, are detected by the dft and deft model as well as mistral and gpt-4o, but not by the cmm model. however, the deft model does not detect the sex-related gebumst (fucked in the sexual sense), while the other two models and gpt-4o accurately assign labels to this word. additionally, it could be expected that the german-only dft model does not detect the english motherfucking, even though the underlying base model was trained on multilingual data, while the cmm model correctly classifies this example. however, it is curious that the codemixing fine-tuned multilingual cmm model fails to detect its literal translation mutterficker, while the other two models label it correctly. all three llms correctly labeled the english and german version of the swear word. neologism detection. in order to compare the models' ability to handle neologisms, their prediction for novel swear words is contrasted with a gold standard list shown in ## discussion this is, to the best of our knowledge, the first proposal of datasets and approaches to combine wordlevel hate speech detection and code-mixing. as real-world and (mostly) manually curated datasets, a benchmark for word-level hate speech detection is provided that also considers code-mixing and neologisms. rap lyrics are particularly prone to novel profanities as well as code-mixing within words, phrases and compounds, making them an especially interesting testbed for hate speech detection and zero-shot domain and language transfer. as can be seen from the results, the inclusion of the languages involved in the code-mixing improves the overall performance of the token classifiers. adding english data to fine-tuning xlm-r on word-level hate speech detection, improved the overall performance on two unseen code-mixed test datasets, one of them even considering several domains. in contrast to the best performing model trained on code-mixed data in @xcite , the german-english token classifier outperformed an identical base model specifically trained on german-english code-mixed data in the presented experiments. this indicates a strong zero-shot domain transfer and a better performance when fine-tuning the same base model on separate english and german examples than on code-mixed examples. however, two performance-relevant factors are the quality and the size of the datasets. the code-mixed model was trained on a subset of ocd, which draws example from twitter and gab, while the other two models were trained on curated datasets from different online domains. the quality of the latter might indeed be high, however, the size of the former is considerably lower than the other two datasets. it is particularly interesting that the xlm-r model trained on multilingual hate speech without code-mixing not only performed better than the code-mixing-specific multilingual model, but also better than three state-of-the-art llms tested in a zero-shot scenario. while two of the llms correctly identified the majority of the novel swear words, their overall detection performance lagged behind the zero-shot performance of the model finetuned on german-english hate speech, indicating a positive impact of language-specific fine-tuning on a code-mixing task. the code-mixed cmm model in this experiment uses the same base model as @xcite and a subset of their dataset, since we needed to annotate the dataset on word-level. even though this smaller subset was used for fine-tuning, the overall f1 scores are better for de-en code-mixed hate speech detection than the previous approach, which could be an indication of better hate speech detection when treating the task as token rather than sentence classification. the results also show that specific categories of hate speech are more challenging for the fine-tuned models than others, especially sex-and genderrelated hate speech sequences. in contrast to previous categorizations (e.g. @xcite , explicit hate speech sequences related to sexual intercourse and sex organs, categorized as sex in this work, turned out to be prevalent in the rap dataset. such sex-related instances might relate to gender in the sense of the intended deformation of specific groups, however, are kept separate for a generally more violent note. the second most frequent category relates to gender-specific instances. hate speech tends to disproportionately target vulnerable groups, such as women, people of color, lgbtq+ individuals, and other marginalized communities, exacerbating social inequalities. addressing this issue is essential to mitigating its impact and ensuring a balance between technological advancement and the protection of these groups. in this approach, the focus is on explicit, codemixed hate speech, which potentially disregards sequences of implicit hate speech that do not explicitly contain swear words. for instance, weißer völkermord is real (white genocide is real) appears in the context of a reference to the world-wide asian population and represents an instance of implicit hate speech. taking implicit hate speech into consideration, especially with a word-level annotation, is an interesting endeavor for future work. to the best of our knowledge, rap lyrics have not been considered for this task. the recognition of rap lyrics as a domain for code-mixed word-level hate speech detection holds substantial significance, not only for german and english, but also for its potential applicability to other languages for future work. this is exemplified by artists, such as capital bra, who incorporate up to 14 different languages in their lyrics, and olexesh, who integrates 11 @xcite , highlighting the complex and multilingual nature of this domain. thus, one fu-ture direction would be to consider the multiplicity of languages in such openly available datasets. ## conclusion hate speech detection has most commonly been treated as a sentence labeling task. instead, this paper addresses it as a word-level annotation and token classification task, providing cross-domain training and test datasets as well as a test set of novel swear word creations. the most central contribution is the consideration of code-mixed dataset for german-english sequences, which in combination with word-level hate speech detection is a novel task and seeks to consider all marginalized groups in this important topic. as a major finding, the inclusion of languages involved in the codemixing for fine-tuning a word-level hate speech detection model improves the results as compared to a model trained on code-mixed data. furthermore, the token classification method improves the results as compared to previous hate speech detection methods on larger datasets with sentence labels. we would, thus, propose word-level hate speech detection as an improved variant of the sentencelevel task, especially for detecting code-mixed hate speech. since multilingualism and code-mixing are increasingly common in online contents, it is high time to consider this aspect in hate speech detection tasks. as regards neologisms, all fine-tuned models perform surprisingly well, only missing two or three novel uses of (code-mixed) swear words. as a main endeavor for future work, we consider the domain of rap lyrics across languages as a valuable resource for code-mixed hate speech datasets. given its fast-paced and dynamic nature, it naturally contains intra-sentential, intra-compound, and intra-word code-mixed hate speech sequences. in addition, this approach only considers explicit hate speech and it would be interesting to extend the approach to annotate and include implicit hate speech, especially in combination with code-mixing.