Length-aware Byte Pair Encoding for Mitigating Over-segmentation in K orean Machine Translation (2024)

Abstract: Byte Pair Encoding is an effective approach in machine translation across several languages. However, our analysis indicates that BPE is prone to over-segmentation in the morphologically rich language, Korean, which can erode word semantics and lead to semantic confusion during training. This semantic confusion, stemming from over-segmentation, ultimately contributes to a degradation of overall translation quality. To address this issue, we introduce Length-aware Subword Vocabulary Construction (LeVoC), a novel approach strategically incorporating longer words into the vocabulary. By utilizing an external monolingual Korean corpus, LeVoC extracts and integrates long words, effectively preserving morphological information and reducing semantic confusion. Our experiments demonstrate that LeVoC not only significantly outperforms BPE, but also can be applied to and surpass current state-of-the-art morpheme-aware subword tokenization methods. We provide evidence that the difficulty in translating sentences with long words in Korean is associated with morphological compositionality, and LeVoC’s ability to reduce semantic confusion during training leads to improved translation quality.

## introduction the utilization of subword-unit segmentation methods has become a prevailing trend in neural machine translation (nmt), with widespread acceptance among researchers @xcite . among the various methods available, byte pair encoding (bpe) @xcite has emerged as a popular choice in numerous nmt studies due to its language-agnostic applicability and its ability to strike a balance between compression efficiency and computational complexity @xcite @xcite @xcite @xcite . although bpe is effective, it often results in over-segmentation when applied to korean, an agglutinative and morphologically rich language @xcite . this issue primarily stems from the language-agnostic and frequency-based characteristics of bpe, which can inadvertently lead to excessive segmentation of words. concretely, we identified three potential instances of over-segmentation, as presented in over-segmentation not only distorts the meaning of sentences but also introduces semantic noise during the training phase, which can significantly degrade the translation quality @xcite . in our investigation, we have observed that this semantic noise is further exacerbated in domain shift scenarios, where the model is challenged with outof-domain test sets. this accumulation of semantic confusion is particularly detrimental in translation tasks where the fidelity of sentence semantics is paramount. our findings underscore the critical need for addressing over-segmentation to ensure robustness against domain shifts and to maintain the integrity of translated content. to address this challenge, we propose levoc (length-aware subword vocabulary construction), a novel approach specifically designed to mitigate the over-segmentation issue in korean. by prioritizing morphologically rich long words during tokenization, levoc effectively alleviates oversegmentation, paving the way for more accurate and meaningful translations. to implement levoc, we first create a large-scale vocabulary using bpe and then build an auxiliary vocabulary set consisting of morphologically rich long words derived from it. we then construct a new tokenizer by incorporating the auxiliary vocabulary into the smallsized bpe vocabulary. more specifically, levoc preserves the existing tokenizer's capacity to overcome the issue of out-of-vocabulary (oov) words, and the integration of long words from the auxiliary vocabulary additionally relieves the tendency for over-segmentation, leading to a more coherent and accurate representation. through our experiments, we demonstrate that levoc mitigates over-segmentation issues and reduces semantic confusion in korean nmt tasks, thereby enhancing translation performance. we demonstrate that the nmt model trained using levoc effectively preserves the morphological structure of the sentences, resulting in superior performance on out-of-domain data that were not present during training. to the best of our knowledge, our method is the first to enhance translation performance by addressing over-segmentation issues stemming from the linguistic characteristics of korean in an unsupervised manner. we publicly release levoc to support further studies foot_0 . ## related works the constraints of bpe @xcite , especially when applied to agglutinative languages with rich morphological features, have been extensively reported. these languages, characterized by the fusion of numerous morphemes into individual words, pose a significant obstacle to the subword tokenization approach, which is fundamentally based on frequency or likelihood of occurrence @xcite . several studies have attempted to address these limitations. for instance, some studies have applied morphological segmentation prior to subword tokenization @xcite , while others have utilized part-of-speech (pos) tagging as additional tag features @xcite . @xcite proposed a method to preserve the morphological structure by recursively exploring the longest sub-string. in addition, some studies have explored alternative approaches to subword tokenization. @xcite introduced a tokenization-free deep encoder that allows parameters to be shared by hashable code points without assigning embeddings for each character. @xcite proposed a method to handle noises robustly by learning and generating the subword embeddings through a pretrained language model without any restrictions on the vocabulary. while numerous studies have explored subword tokenization, the issue of over-segmentation specific to korean language processing remains largely unexplored. levoc mitigates over-segmentation through a focus on vocabulary construction particularly in the context of the korean language. ## qualitative analysis segmentation and translation fidelity in levoc addresses this issue by prioritizing including longer, morphologically rich subwords into the vocabulary, which supports the learning process by providing a clearer semantic signal. ## conclusion in this paper, we focus on the over-segmentation issue in the korean mt models that erodes morphological information in the text. we discovered that conventional bpe tokenizers, while effectively alleviating the oov problem, are prone to oversegmentation. we found that this semantic confusion, which stems from over-segmentation, ultimately degrades the overall translation quality. to address this issue, we propose levoc, a simple and effective method for mitigating over-segmentation by preserving the morphological structure of long tokens. we found that levoc can notably enhance the mt performance and preserve morphological information, especially for morphologically rich and ideal settings. levoc can be seamlessly integrated with a morpheme analyzer designed for the korean language to maintain morphological integrity. our comprehensive evaluation revealed that addressing over-segmentation via manipulating token length distribution in a bpe vocabulary improves mt performance substantially. we hope that the findings from our study will inspire further research in natural language processing by considering korean's unique characteristics.