K- MH a S : A Multi-label Hate Speech Detection Dataset in K orean Online News Comment (2022)

Abstract: Online hate speech detection has become an important issue due to the growth of online content, but resources in languages other than English are extremely limited. We introduce K-MHaS, a new multi-label dataset for hate speech detection that effectively handles Korean language patterns. The dataset consists of 109k utterances from news comments and provides a multi-label classification using 1 to 4 labels, and handles subjectivity and intersectionality. We evaluate strong baselines on K-MHaS. KR-BERT with a sub-character tokenizer outperforms others, recognizing decomposed characters in each hate speech class.

## introduction the growth of online content including social media @xcite , news comments @xcite , wikipedia @xcite , and in-game chat @xcite presents challenges in detecting hate speech using advanced natural language processing. hate speech is language that attacks or diminishes individuals or groups based on certain characteristics such as physical appearance, religion, gender, or other attributes, and it can occur across different linguistic styles @xcite . hate speech detection is intrinsically a complex task @xcite due to the fuzzy boundary with other overlapping concepts such as abusive language @xcite , toxic comments @xcite , or offensive language @xcite . recently, the rise in popularity of korean tv, movies, and music (e.g. squid game, bts) has led to many young people showing an interest in learning korean. this phenomenon could result in exposure to harmful content and hate speech in korean. however, (1) the most common language in hate speech research is english and only limited resources are available in other languages such as arabic @xcite , dutch @xcite , and korean @xcite . in addition, most datasets are annotated (2) using a single label classification of particular aspects, even though the subjectivity of hate speech cannot be explained with a mutually exclusive annotation scheme. we propose k-mhas, a korean multi-label hate speech detection dataset that allows overlapping labels associated with intersectionality, a concept from sociology that identifies combined attributes @xcite . our dataset consists of 109,692 utterances from korean online news comments, labeled with 8 fine-grained hate speech classes. k-mhas is compatible with previous work on hate speech in other languages, by providing binary classification and multi-label classification from 1(one) to 4(four) labels. we investigate the k-mhas dataset by analyzing label distribution, keywords, and label pairs. in addition, we provide strong baseline pre-trained language models using multilingual-bert, ko-electra, kobert, and kr-bert, and compare the results using six metrics for multi-label classification tasks. overall, the koelectra model achieves the best performance for all labels, indicating the effects of the pre-training data source. the kr-bert with a sub-character-level tokenizer outperforms the others on several label pairs, showing that decomposing various korean characters is essential for the task. our contribution can be summarized as follows: publication language source data size labels m-label waseem and hovy (2016) english twitter 16.2k sexism, racism, neither n davidson et al. (2017) english twitter 24.8k hate speech, offensive, neither n wulczyn et al. (2017) english wikipedia comments 115k toxic, severe toxic, obscene, threat, insult, identity hate, neutral y ibrohim and budi (2019) indonesian twitter 11k (a) individual, group (b) religion, race, pysical, gender, other (c) weak, moderate, strong hate speech p fortuna et al. (2019) portuguese twitter 5.6k (a) hate speech, not hate speech (b) sexism, body, origin, homophobia, racism, ideology, religion, health, other-lifestyle p ousidhoum et al. (2019) english french arabic twitter 6k (en) 4k (fr) 3k (ar) labels for five different aspects (a) directness, (b) hostility, (c) target, (d) group, and (e) annotator p moon et al. (2020) korean news comments 9k (a) hate speech, offensive, none (b) gender, others, none n ours korean news comments 109k (a) hate speech, not hate speech (b) politics, origin, physical, age, gender religion, race, profanity, not hate speech y ## korean multi-label hate speech detection dataset (k-mhas) our dataset is based on the korean online news comments available on kaggle 2 and github 3 . the unlabeled raw data was collected between january 2018 and june 2020. in order to curate the data, we randomly select more than 109,692 news comments. our data preprocessing is designed to tokenize a korean character and to filter the length. we remove urls and bad characters (e.g. u+1100 to u+11ff -hangul jamo) using regular expressions while keeping uppercase and lowercase letters in english and emoji. we discard sentences with fewer than 10 characters as it is often only one word. for the data derived from online comments, we normalized repeated characters by truncating their number of consecutive repetitions to two. ## dataset analysis label distribution the keyword analysis is from the total dataset and different from some examples in annotation guidelines. mon in other hate speech datasets. this difference could be because korea is a highly homogenous monoculture with little variation in race and religion @xcite . interestingly, the 'gender' class (16.3%) occurs at almost twice the frequency in a multi-label distribution, compared to a single label distribution (9.2%). this indicates that gender-based hate speech is used extensively in combined aspects. ## results evaluation for multi-labels koelectra, outperforms other multi-label tasks due to domain similarity. for the multi-label classification, kr-bert-s achieved the best performance. it uses a sub-character tokenizer that can decompose hangul(korean language) syllable characters into sub-characters. therefore, it provides greater granularity in detecting hate speech words, by identifying the sub-characters from different hate speech categories. 8 . evaluation for label-pairs ## conclusion we propose k-mhas, a new large-sized dataset for korean hate speech detection with a multi-label annotation scheme. we provided extensive baseline experiment results, presenting the usability of a dataset to detect korean language patterns in hate speech. in future work, the automatic hate speech moderation and counter-speech can be expanded. state-of-the-art natural language processing. in proceedings of the 2020 conference on emnlp: system demonstrations, pages 38-45. association for computational linguistics. ellery wulczyn, nithum thain, and lucas dixon. 2017. ex machina: personal attacks seen at scale. in proceedings of the 26th international conference on world wide web, pages 1391-1399. marcos zampieri, preslav nakov, sara rosenthal, pepa atanasova, georgi karadzhov, hamdy mubarak, leon derczynski, zeses pitenis, and çagrı çöltekin. 2020. semeval-2020 task 12: multilingual offensive language identification in social media (offenseval 2020). in proceedings of the fourteenth workshop on semantic evaluation, pages 1425-1447.