COLD : A Benchmark for C hinese Offensive Language Detection (2022)

Abstract: Offensive language detection is increasingly crucial for maintaining a civilized social media platform and deploying pre-trained language models. However, this task in Chinese is still under exploration due to the scarcity of reliable datasets. To this end, we propose a benchmark –COLD for Chinese offensive language analysis, including a Chinese Offensive Language Dataset –COLDATASET and a baseline detector –COLDETECTOR which is trained on the dataset. We show that the COLD benchmark contributes to Chinese offensive language detection which is challenging for existing resources. We then deploy the COLDETECTOR and conduct detailed analyses on popular Chinese pre-trained language models. We first analyze the offensiveness of existing generative models and show that these models inevitably expose varying degrees of offensive issues. Furthermore, we investigate the factors that influence the offensive generations, and we find that anti-bias contents and keywords referring to certain groups or revealing negative attitudes trigger offensive outputs easier.

## introduction offensive language detection task plays an essential role in maintaining social platforms and promoting civilized communication @xcite @xcite . with the rise of large-scale language models @xcite @xcite , the safety issues due to offensive generation continue to be exposed @xcite @xcite , attracting widespread attention from researchers and pushing the research boom on this task to new heights @xcite @xcite . to tackle the problem of offensive language detection, a reliable and versatile benchmark is a needed basis to accelerate in-depth research. the datasets, including wtc @xcite , olid @xcite , bad @xcite and @xcite , are proposed to study the safety issues from different dimensions and granularities. the publicly available detector, perspectiveapi foot_1 , is widely used for toxicity evaluation and contributes to creating safer environments for online communication @xcite . however, most existing works focus on english. the issue of chinese offensive language detection has not been well studied due to the lack of labeled datasets and reliable detectors. in addition, large-scale language models often lean biases in pre-training data and generates offensive or unethical contents @xcite , which substantially hinders the deployment of models in practice @xcite . meanwhile, limited by reliable benchmarks, the offensiveness of chinese language models has not yet been thoroughly studied. how offensive can chinese language models be? what contents influence the triggering of offensive generation? diving deeper into these questions will facilitate building more reliable and deployable language models. this paper proposes a benchmark named cold to tackle the above challenges in chinese offensive language research. the coldataset (chinese offensive language dataset), contains 37,480 comments with binary offensive labels and covers diverse topics of race, gender, and region. to gain further insights into the data types and characteristics, we annotate the test set at a fine-grained level with four categories: attacking individuals, attacking groups, anti-bias and other non-offensive. we present a baseline detector, coldetector, for offensive language detection, which adopts pretrained chinese bert and is fine-tuned on the proposed dataset and performs satisfactorily compared to other methods using existing resources and technology. with the proposed benchmark cold, we evaluate the offensiveness of popular chinese generation models, including cdialgpt @xcite , cpm @xcite , and eva @xcite , to investigate their strengths and weaknesses in terms of safety. experimental results show that both offensive and non-offensive inputs have the risk of inducing safety issues.additionally, some types of prompts, including anti-bias contents, certain target group keywords and negative attitude words, can more easily trigger offensive outputs than other inputs. figure 1 the contributions of this work are threefold: ## dataset construction we present coldataset, a chinese dataset containing 37k sentences and covering the topics of racial, gender, and regional bias. our data collection process is in line with the suggestions provided by @xcite to achieve standardized and accountable research benchmarks. ## offensive language detection the experiments of offensive language detection are designed to verify the following two questions: can offensive language be detected with existing resources and technology alone? does proposed coldataset effectively advance the offensive language detection task? ## evaluation of generative lms with the proposed coldataset and colde-tector, we evaluate the offensiveness of popular chinese generative language models. we mainly investigate the following research questions. rq1: how offensive are the chinese generative language models? rq2: what type of prompts can trigger offensive generation? ## conclusion we present a new dataset named coldataset for chinese offensive language analysis. we show that the proposed coldetector trained on our data can effectively detect offensive content. it can also be used as a benchmark for the offensiveness evaluation of language models. we evaluate some popular used models and reveal that they have different degrees of risk in generating offesive contents. besides, our work shows that, for language models, non-offensive input can also induce safety problems as offensive input, and is worth the same attention. in particular, anti-bias language, which is non-offensive but has hazards comparable to offensive input, is often overlooked in existing work. we hope this new benchmark can provide the basis for safety research in chinese and shed light on further studies. we call for more research to expand the scope of offensive and other unsafe language detection. besides, we believe that, further investigating on what types of input successfully induce unsafe generation, will facilitate the safer deployment of language models.