KOAS : K orean Text Offensiveness Analysis System (2021)

Abstract: Warning: This manuscript contains a certain level of offensive expression. As communication through social media platforms has grown immensely, the increasing prevalence of offensive language online has become a critical problem. Notably in Korea, one of the countries with the highest Internet usage, automatic detection of offensive expressions has recently been brought to attention. However, morphological richness and complex syntax of Korean causes difficulties in neural model training. Furthermore, most of previous studies mainly focus on the detection of abusive language, disregarding implicit offensiveness and underestimating a different degree of intensity. To tackle these problems, we present KOAS, a system that fully exploits both contextual and linguistic features and estimates an offensiveness score for a text. We carefully designed KOAS with a multi-task learning framework and constructed a Korean dataset for offensive analysis from various domains. Refer for a detailed demonstration.

## introduction online communities and social media have become the mainstream platforms of communication. this has also led to unwanted developments -an increasing use of offensive language through online platforms. consequently, analyzing texts and detecting offensive expressions has become a critical issue @xcite . however, manual detection of offensive texts is infeasible owing to the increasing popularity of social networks @xcite . notably in south korea, high internet accessibility and social media usage 2 have stimulated a dire need for a system that analyzes korean text and its offensiveness @xcite . despite the recent success of offensive language detection on english text @xcite , handling figure 1: (a) illustration of an example of offensive sentence without any explicit profanity, which would be classified as "non-abusive" in abusive language detection. (b) illustration of sentences with different intensity of offensiveness that is not distinguished properly in the discrete classification tasks. korean texts is quite challenging. owing to the highcontext cultural characteristics of korean language culture @xcite , korean offensive expressions tend to be expressed in a subtle and figurative way without explicit abusive expressions as illustrated in figure 1(a) . additionally, large vocabulary and the complex syntax as a morphologically rich and agglutinative language @xcite often hinders the model's learning @xcite . another substantial problem in text analysis is that the intensity of offensiveness in text is often neglected. as shown in figure 1(b) , the sentence may address the different intensity depending on the degree of frequency and explicitness of the offensive expression @xcite . however, most researches focus on simply classifying sentences into discrete, sometimes binary, categories @xcite @xcite and treat sentences with different intensity as the same type, "negative", or "abusive" for instance. in this demo, we present koas, a system that estimates the score of offensiveness in korean text. since the degree of offensiveness is different from of abusive detection or sentiment analysis, we develop a scoring function for offensiveness. the score is to quantify how much negative feelings each sentence can cause to readers, or how offensively it can be read. to this end, koas internally conducts two classification tasks, abusive language detection and sentiment analysis. an offensiveness score is then elicited from the outputs of two internal tasks. while computing the score, koas integrates the semantic perspective where it detects explicit abusive expressions and semantic perspective where it captures implicit nuance and tone of the text. since the notion of continuous degree of offensiveness has not been researched in depth yet, there are no appropriate datasets to evaluate offensiveness score. therefore, we construct new datasets for the abusive language detection and sentiment classification and utilize them for evaluating our systems for the offensiveness analysis. to handle korean data, we utilize a refined morpheme-level tokenization method, which has an effect of data augmentation and subword regularization. we summarize our contributions as follows. ## system design the architecture of koas is shown in figure 2 . the system mainly consists of two parts: a classification model and an offensiveness scoring function. we will describe them in detail in the following subsection. model architecture the overall model architecture was inspired by the gated multi-task learning framework @xcite . we denote the proposed neural network as cnn m t l and compare it with the baseline of the pipelined vanilla cnn @xcite . cnn m t l jointly learns two text analysis tasks -abusive language detection and sentiment analysis. the model is trained with two tasks, learning both linguistic perspective and semantic perspectives simultaneously. this leads to a more computationally efficient model with better performance than training two separate models for each task. utilizing task-specific layers as well as a shared layer has proven to be effective in learning not only task-dependent features but also useful common features @xcite @xcite . the neural model of koas consists of an embedding layer, two task-specific convolution layers for the sentiment analysis and abusive language detection, a shared convolution layer, and two softmax layers. the embedding layer transforms an arbitrary-length input sentence into a matrix of embeddings, denoted as @xmath0... ⊕ w s , where @xmath1, ⊕ denotes the concatenation operator, and s and k are the number of words and the dimension of word embedding, respectively. then, the embedding matrix is fed into the three convolution layers. the features h se , h ab , h sh are obtained from the convolution layers. all the convolution layers are followed by relu, max pooling, and dropout layers. the features are then concatenated and fed into the softmax layers. additionally, we employ a gate mechanism @xcite when incorporating task-private convolution output to the other task. we introduce two gates g s2a and g a2s , which control the flow of features from the sentiment analysis task to the abusive language detection task and vice versa, respectively. the gates share useful features and prevent irrelevant information from being propagated. this is calculated by text offensiveness the score for text offensiveness is based on the following rules: therefore, negative tones and abusive expression in the text have a positive correlation with offensiveness, whereas a positive tone has a negative correlation. based on these hypotheses, we propose a new method for the quantification of offensiveness: ## demonstration koas has an intuitive and simplified interface, where users can type any korean sentence and check the offensiveness of the input sentence. when koas receives a sentence, it internally conducts abusive language detection and sentiment analysis, and then computes the offensiveness score. the logit value of the sentence containing an abusive expression and the most likely sentiment polarity is shown at the bottom, as well as the offensiveness score. figure 3 shows a user's input interface of koas. there is a status bar in the middle of the screen, so users can check the level of offensive intensity. when the score is higher than a predefined moderate threshold, messages like "this comment might have malicious intent." and crucial threshold, warning messages like "this comment is malicious and at the risk of being deleted." appear on the screen. in this work, we set the moderate threshold to be 40 and crucial threshold to be 72. ## conclusion in this paper, we have proposed koas, a novel system that efficiently estimates the offensiveness score of korean text. we expect koas to attenuate the usage of offensive languages by providing a real-time feedback to writers about their writing. koas has notable technical novelty and social contributions, including (1) tackling morphological richness and complex syntax of korean, (2) incorporating linguistic and contextual analysis to capture the offensive nuance of text and (3) effectively analyzing the offensiveness of text. our cnn-based system is lightweight, practical and applicable to various hardware environments compared to transformer-based system. some usecases we expect are as follows: first, for the people unfamiliar with korean, the koas system can be used to prevent unintended attacks when they post korean articles and help them recognize attacks in korean sites. second, site administrators who need to block offensive comments can port the koas system to automatically block comments that exceed a certain score. finally, our datasets can be utilized for further research on korean text analysis, including korean language understanding and automatic labeling. we expect that our proposed system will be readily applicable to various downstream applications, including education, game, real-time chatting and social media platforms.