T oxi C loak CN : Evaluating Robustness of Offensive Language Detection in C hinese with Cloaking Perturbations (2024)

Abstract: Detecting hate speech and offensive language is essential for maintaining a safe and respectful digital environment. This study examines the limitations of state-of-the-art large language models (LLMs) in identifying offensive content within systematically perturbed data, with a focus on Chinese, a language particularly susceptible to such perturbations. We introduce ToxiCloakCN, an enhanced dataset derived from ToxiCN, augmented with homophonic substitutions and emoji transformations, to test the robustness of LLMs against these cloaking perturbations. Our findings reveal that existing models significantly underperform in detecting offensive content when these perturbations are applied. We provide an in-depth analysis of how different types of offensive content are affected by these perturbations and explore the alignment between human and model explanations of offensiveness. Our work highlights the urgent need for more advanced techniques in offensive language detection to combat the evolving tactics used to evade detection mechanisms.

## introduction offensive language, which includes hate speech, cyberbullying, and adult-oriented content, poses significant risks to user well-being and social harmony @xcite . with the rapid expansion and widespread usage of social media platforms, the proliferation of offensive language has become a critical issue. consequently, social media platforms and researchers have explored developing robust machine learning and linguistic analy-* yunze xiao and yujia hu contributed equally to this work. 1 github: https://github.com/social-ai-studio/ toxicloakcn sis solutions to effectively identify and mitigate the harmful effects of offensive content @xcite . recent advances in natural language processing (nlp), particularly with large language models (llms), have significantly improved the ability to detect offensive language across multiple languages @xcite @xcite @xcite @xcite @xcite . however, these models often struggle with systematically perturbed data designed to evade detection mechanisms. common perturbation techniques include homophonic substitutions, emoji replacement, insertions, character splits, and synonyms @xcite . these techniques, referred to as "cloaking", exploit linguistic nuances to mask offensive content, posing a substantial challenge to both automated systems and human moderators. the chinese language, in particular, is heavily impacted by these techniques due to intensive lexiconbased censorship, leading to a new linguistic phenomenon @xcite where significant parts of sentences are replaced by either homophones or emojis to mask underlying offensive content or to circumvent censorship rules. figure 1 shows two examples of offensive texts cloaked using homophone and emoji replacement techniques. in these examples, the words and phrases highlighted in yellow are replaced with homophones or emojis. in the first example, homophones are used to replace phrases that identify the target (e.g., "贺楠 仁" as the homophone for "河南人," which means people from the henan region in china) and offensive terms such as "太贱" with "肽键." similarly, in the second example, the offensive term "舔狗" (i.e., simps) is replaced with . using such techniques, users can fool automated offensive lan-figure 1 : example of cloaked chinese offensive language using homophone and emoji replacement. by using such techniques, users will be able to fool the automated offensive language detector into misclassifying them as normal sentences. in this work, we introduce toxicloakcn, a novel chinese offensive content dataset that benchmark content moderation models' ability to detect offensive texts cloaked using homophone and emoji replacements. specifically, we conduct extensive experiments and evaluate state-of-the-art llms on the toxicloakcn dataset. the experiments demonstrated that both perturbation methods significantly affect the models' capabilities in detecting offensive text. we also analyze the effect of prompts on the experimental results by testing the models using six different prompts. additionally, we analyze the perturbation effects on different types of offensive content: sexism, racism, regional bias, and anti-lgbtq+. this research underscores the critical need for developing more robust models to effectively moderate cloaked online offensive content. we summarize the main contributions of this paper as follows: ## methodology the toxicloakcn dataset builds upon the tox-icn dataset @xcite through a detailed multi-step process. first, we sampled a balanced dataset from the base toxicn dataset, known as the "base" dataset. next, this balanced base dataset was perturbed using homophone and emoji replacements to produce the toxicloakcn dataset. for such after constructing the toxicloakcn dataset, we explored pinyin augmentation as a potential solution to address the "cloaked" offensive content perturbed using homophone replacements. finally, we defined six different instructions for evaluating the performance of state-of-the-art large language models on toxicloakcn. ## conclusion and future works this study investigated the robustness of chinese offensive language detection models against cloaking perturbations, specifically homophone and emoji replacements. we developed the tox-icloakcn dataset by augmenting the toxicn dataset with these perturbations to simulate realworld evasion tactics. our experiments showed that state-of-the-art models, including gpt-4o, experienced significant performance drops when encountering cloaked offensive content. while our proposed pinyin augmentation method showed some promise, its effectiveness varied across models, underscoring the complexity of phonetic alignment in offensive language detection. case studies further revealed gaps in model comprehension of cloaked offensive content compared to human annotators. gpt-4o frequently missed or misinterpreted offensive words disguised with homophones or emojis, whereas human evaluators, aided by cultural and contextual knowledge, identified the offensive nature of the texts accurately. this highlights the need for models that better mimic human understanding of nuanced, contextrich language and emphasizes the urgency of developing more advanced techniques to address evolving evasion strategies. future research should explore cloaking techniques beyond homophones and emojis, incorporate broader linguistic variations from real-world internet sources, and develop more sophisticated phonetic alignment methods to enhance model robustness. additionally, integrating deeper semantic understanding and context-awareness into algorithms will be critical for effectively managing cloaked offensive language. given the broader relevance of this phenomenon, future work should extend these methods to a multilingual setting. addressing these areas can significantly advance offensive language detection, contributing to safer digital environments. any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the ministry of education, singapore.