The K orean Morphologically Tight-Fitting Tokenizer for Noisy User-Generated Texts (2021)

Abstract: User-generated texts include various types of stylistic properties, or noises. Such texts are not properly processed by existing morpheme analyzers or language models based on formal texts such as encyclopedias or news articles. In this paper, we propose a simple morphologically tight-fitting tokenizer (K-MT) that can better process proper nouns, coinages, and internet slang among other types of noise in Korean user-generated texts. We tested our tokenizer by performing classification tasks on Korean user-generated movie reviews and hate speech datasets, and the Korean Named Entity Recognition dataset. Through our tests, we found that K-MT is better fit to process internet slangs, proper nouns, and coinages, compared to a morpheme analyzer and a character-level WordPiece tokenizer.

## introduction user-generated texts such as internet forums, product reviews, blog posts, tweets, and comments include various types of noise coming from their stylistic properties. noise such as internet slang, spelling errors, and emojis may not be properly processed by traditional morpheme analyzers and language models, since such models are trained on formal texts such as books, encyclopedias, or news articles. user-generated texts that include such properties are useful to analyze public opinion on a variety of topics, products, policies, and so on. noise in texts may vary in their forms with different languages, even though they are common to some degree. for example, emphasized opinion can be written capitalized in english, while many exclamation marks are used in korean as in an example: 하지마hacima!!!!!!!! "don't do it!". such noise has to be processed, rather than removed, depending on the purpose. accordingly, we analyzed various korean usergenerated texts and obtained an unexhaustive list of noise types observed in such texts: spacing errors, spelling errors, grammatical errors, special characters and emojis, korean grapheme characters, foreign language characters, newly-listed proper nouns, coinage, and internet slang. korean grapheme characters such as 'ㅋㅋ,' 'ㅠㅠ' are often used and function as emojis. in this paper, we focus on proper nouns, coinages, and internet slang among other types of noise from user-generated texts. we propose a korean morphologically tight-fitting tokenizer, k-mt, which is expected to be robust in processing such types of noises. we compare the performance of our k-mt with a korean morpheme analyzer and a character-based wordpiece tokenizer @xcite through experiments on korean movie reviews, a hate speech dataset, and a named entity recognition dataset. our major contributions are: ## experiments and analysis to evaluate our k-mt, we compare the tokenized results with those of mecab-ko, a korean morpheme analyzer, and a character-based subword tokenizer, wordpiece. mecab-ko is an upper bound model in this paper, and our goal is to construct a simpler tokenizer imitating the elaborated morpheme analyzer as possible. first, we selected two popular noisy, usergenerated datasets: naver sentiment movie corpus (nsmc) and hate speech dataset (hsd). both datasets include most of the noise types we briefly mentioned in section 1. nsmc includes binary sentiment polarity labels, positive and negative, on each film review and consists of 13k, 2k, and 5k reviews for training, validation, and test set. hsd consists of 7,896, 471, and 974 news comments for training, validation, and test set, respectively. we performed two different tasks on hsd: bias classifi-cation (whether a comment includes 'gender' bias, 'other' types of bias, or 'none') and hate speech detection (whether a comment includes 'hate' speech, less hateful but 'offensive' speech, or 'none'). for the three tasks, we applied a simple 3-layer cnn model. we limited the size of vocabulary for all the tokenizers to 25,000 and used a batch size of 64, an embedding dimension of 128, 128 convolutional filters, and a dropout of 0.2. we used an adam optimizer and cross-entropy loss and trained the model for 10 epochs. in the two tasks on the hsd dataset, the three tokenizers show larger differences in performance, where mecab-ko, the upper bound model, reports the best accuracies. while the wordpiece tokenizer generally obtained higher accuracies than k-mt, k-mt performed better than wordpiece in hate speech detection. in addition, we performed a named entity recognition task with the three tokenizers above. we used the publicly released training set foot_6 from the naver nlp challenge 2018, dividing it into 76,949, 4,050, and 8,999 sentences for each of the training set, validation set, and test set. the named entity labels include 14 types of named entities: person, field, organization, location, artificial objects, cultural terms, date, time, number, event, animal, plant, material, other terms, and none. the ner dataset includes various named entity words and rarely includes other types of noises like spacing errors. we included this experiment since our k-mt is expected to work well in separating and preserving proper nouns and named entities which are likely to be newly-encountered domainspecific words in texts. we applied a simple 2-layer rnn model. we used a batch size of 64, an embedding dimension of 100, a hidden dimension of 128, and a dropout of 0.25. we used an adam optimizer and cross-entropy loss and trained the model for 10 epochs. when we first limited the size of vocabulary for all the tokenizers as 25,000, we observed the best accuracies of 67.57 and 68.02 for validation and test set with wordpiece tokenization, and similar performance with mecab-ko. our k-mt had marginally lower accuracies of 63.02 and 63.37 for the validation and the test set, respectively. on the other hand, we used unlimited tokens for the models' vocabulary with mecab-ko and k-mt to remove the effect of the out-of-vocabulary problem. then we obtained higher accuracies; especially, k-mt reported a validation accuracy of 72.78 and a test accuracy of 73.05, which are higher than mecab-ko. this indicates the potential of our tokenizer to be robust to newly-listed terms, proper nouns, and named entities in user-generated texts. to analyze the performance of k-mt, we took 500 random samples from each of the nsmc, hsd, and ner datasets for tokenization. first, we quantitively compared the tokenization of the different tokenizers. in general, our k-mt is good at tokenizing words in korean including: since nsmc and hsd datasets include various types of noise, especially spacing errors, there may have been examples where our k-mt could not accurately tokenize text. moreover, k-mt is not advantageous when dealing with texts of various domains because it uses a limited size of vocabulary without subword tokens, unlike wordpiece. however, k-mt works well with newly-listed words and terms as in the ner dataset, which existing language models based on formal texts cannot properly tokenize. for example, moreover, k-mt is based on a simpler algorithm than mecab-ko and other morpheme analyzers, making it easier to combine with other kinds of recent tokenization models such as the wordpiece model. ## conclusion we implemented a simple korean morpheme-like tokenizer, k-mt, to deal especially with proper nouns, coinages, internet slangs, and newly-listed domain-specific words in user-generated texts. to evaluate the tokenizer, we performed classification tasks on movie reviews, comments on news articles, and a named entity recognition dataset, and we compared the results with those of a morpheme analyzer and character-based wordpiece model. from the results, we observed that our k-mt works well with proper nouns, coinages, and internet slang, while it still needs to be improved for cases with spacing errors. for future work, we will improve our tokenizer by considering context information beyond the word segment boundary and additional methods to deal with the spacing errors. also, we plan to construct a better tokenization algorithm by combining k-mt with the wordpiece model as a hybrid method.