An Empirical Study of Tokenization Strategies for Various K orean NLP Tasks (2020)

Abstract: Typically, tokenization is the very first step in most text processing works. As a token serves as an atomic unit that embeds the contextual information of text, how to define a token plays a decisive role in the performance of a model. Even though Byte Pair Encoding (BPE) has been considered the de facto standard tokenization method due to its simplicity and universality, it still remains unclear whether BPE works best across all languages and tasks. In this paper, we test several tokenization strategies in order to answer our primary research question, that is, “What is the best tokenization strategy for Korean NLP tasks?” Experimental results demonstrate that a hybrid approach of morphological segmentation followed by BPE works best in Korean to/from English machine translation and natural language understanding tasks such as KorNLI, KorSTS, NSMC, and PAWS-X. As an exception, for KorQuAD, the Korean extension of SQuAD, BPE segmentation turns out to be the most effective. Our code and pre-trained models are publicly available at https://github.com/kakaobrain/kortok .

## introduction tokenization is the very first step in most text processing works. not surprisingly, tremendous academic efforts have been made to find the best tokenization method for various nlp tasks. for the past few years, byte pair encoding (bpe) @xcite has been considered the de facto standard tokenization technique since it was reintroduced by @xcite . besides the fact that bpe turns out to be very effective in the machine translation task, another important reason bpe has gained such popularity is that bpe is a data-driven statistical algorithm so it is independent of language. however, it is still not clear whether bpe works best across all languages, irrespective of tasks. in this paper we study various tokenization strategies for korean, a language which is morphologically by far richer than english. concretely, we empirically examine what is the best tokenization strategy for korean to english / english to korean machine translation tasks, and natural language understanding (nlu) tasks-machine reading comprehension (mrc), natural language inference (nli), semantic textual similarity (sts), sentiment analysis, and paraphrase identification. we are particularly interested in how complementary bpe and linguistically motivated segmentation are. ## related work there have been extensive studies about tokenization techniques for machine translation. several papers claimed that a hybrid of linguistically informed segmentation and a data-driven method like bpe or unigram language modeling performs the best for non-english languages. @xcite combined an off-the-shelf morphological segmenter and bpe in hindi and bengali translations against english. @xcite used a retrained version of linguistically motivated segmentation model along with statistical segmentation methods for arabic. @xcite adopted linguistic guidance to bpe for english-latvian translation. particularly @xcite ) is close to ours, but their main focus is on preprocessing techniques for neural machine 2 https://github.com/sktbrain/kobert translation like parallel corpus filtering rather than on tokenization strategies per se. compared with the tokenization studies for machine translation, those for nlu tasks have gained less attention. among them is @xcite , which compared the fine-tuning task performance of bert @xcite pre-trained with bpe and unigram language modeling. moon and okazaki (2020) proposed a novel encoding method for korean and showed its efficiency in vocabulary compression with a few korean nlu datasets. ## tokenization strategies we introduce assorted korean tokenization strategies arranged from the smallest to the largest unit. each of them induces different tokenization results, as illustrated in ## discussion we further examine which factors with respect to tokenization affect the ko-en and en-ko translation performance. ## conclusion we explored various korean tokenization strategies on machine translation and five nlu tasks. in machine translation morpheme-aware subword models with a vocabulary size worked best for both korean to english and english to korean settings. by contrast, there was no single best tokenization strategy for the nlu tasks. instead, subword 64k models showed the best performance on korquad, whereas morpheme-aware subword 64k models turned out to be optimal for the other kornli, ko-rsts, nsmc, and paws-x tasks.