Should we find another model?: Improving Neural Machine Translation Performance with ONE -Piece Tokenization Method without Model Modification (2021)

Abstract: Most of the recent Natural Language Processing(NLP) studies are based on the Pretrain-Finetuning Approach (PFA), but in small and medium-sized enterprises or companies with insufficient hardware there are many limitations to servicing NLP application software using such technology due to slow speed and insufficient memory. The latest PFA technologies require large amounts of data, especially for low-resource languages, making them much more difficult to work with. We propose a new tokenization method, ONE-Piece, to address this limitation that combines the morphology-considered subword tokenization method and the vocabulary method used after probing for an existing method that has not been carefully considered before. Our proposed method can also be used without modifying the model structure. We experiment by applying ONE-Piece to Korean, a morphologically-rich and low-resource language. We derive an optimal subword tokenization result for Korean-English machine translation by conducting a case study that combines the subword tokenization method, morphological segmentation, and vocabulary method. Through comparative experiments with all the tokenization methods currently used in NLP research, ONE-Piece achieves performance comparable to the current Korean-English machine translation state-of-the-art model.

## introduction recent studies using pretrain-finetuning approach (pfa) technique have achieved state-of-the-art (sota) performance in many natural language processing (nlp) tasks and are becoming the latest trend @xcite @xcite @xcite . to utilize the pfa, a large amount of pre-training data and a system with sufficient computing power are required. for example, t5 @xcite was trained with 11 b parameters and 1 t tokens in order to get sota performance, and for gpt3 @xcite , 170 b parameters were required to train a model to demonstrate the best performance. the trend of model research based on pfa raises two problems. first, it is hard to expect a similar performance for the low-resource setting. this is because most studies based on the pfa technique rely on large amounts of data @xcite . but for low-resource languages, it is difficult to provide the comparable amount of data required by recent papers. second, it is necessary to overturn the existing model and pre-train a new model from scratch to create a pfa-based model that follows the latest research trends. since the pfa-based model requires many parameters, companies without adequate server or graphic processing unit (gpu) environments may have many difficulties in configuring the service environment and utilizing the latest model @xcite . therefore, new approaches are required to ensure high performance for low-resource languages and companies lacking extensive server and gpu environments. to solve this problem, many researches are being conducted on the way of improving the performance of nlp application software without changing the model through data pre and post-processing, typically in machine translation @xcite @xcite @xcite . reflecting this trend, we conducted a study on an optimized tokenization that can improve the performance of neural machine translation (nmt) without changing the model. we propose two perspectives for optimized tokenization. first, we analyze the limitations of byte pair encoding (bpe) @xcite and sentencepiece @xcite , which can easily be applied to various languages. due to its language-agnostic characteristic, these methods are currently used as the defaults in language model research and existing tokenization methods. however, there are 7,111 languages around the world. more than 50 million people speak 25 languages as their mother tongue that have various morphological characteristics such as isolating language, agglutinative language, and fusional language. considering this, it seems hard to assert that applying sentencepiece and bpe always produce the best performance. second, we focus on the problem that there is not enough discussion about the corpus used in tokenizer training. several studies that applied bpe and sentencepiece use a merged bilingual corpus, that combines two language corpora into one, when training its tokenizer @xcite . however in these studies, merged bilingual corpus is utilized without sufficient comparative analysis. in this study, tokenization methods which leveraging merged bilingual corpora and separate bilingual corpora are denoted as vocabulary communicating (vc) and vocabulary separating (vs), respectively. we denote vc and vs as vocabulary methods and compare the performance of each method in nmt. in other words, we further figure out the optimal tokenization method through comparative experiments on various tokenization methods. all the experiments are made on a korean dataset, which is a representative of low-resource and morphologically rich language (mrl). in particular, we propose one-piece that combines the vc method and morphological segmentation followed by sentencepiece. through comparative experiments with tokenization methods currently used in nlp research, such as bpe and sentencepiece, we revealed that one-piece can encourage the optimal performance in korean-english machine translation. the contributions of our study are as follows: ## proposed method this study proposes an optimal tokenization method for improving machine translation performance from the viewpoints of morphological segmentation and vocabulary method. we derive an optimal tokenization method for korean-english machine translation by conducting a case study that combines the morphological segmentation and vocabulary method. ## conclusion in this study, we proposed a new tokenization method called one-piece. this can provide the best performance in korean-english machine translation compared with other tokenization methods. our results quantitatively confirmed the effect of the vocabulary method and morphological segmentation on nmt performance. furthermore, we experimentally proved that the vc method and morphological segmentation cannot consistently improve the performance of nmt by themselves. our results showed that significant and consistent performance improvement can only be achieved in nmt if they are properly used together. by using one-piece, the vanilla transformer model shows comparable translation performance to the mbart. accordingly, we expect that companies that have difficulties using the latest pfa-based model, due to an inadequate server environment, will be able to utilize our proposed model to provide sufficiently good performance.