Finding the Optimal Byte-Pair Encoding Merge Operations for Neural Machine Translation in a Low-Resource Setting (2024)

Abstract: This paper investigates the impact of different Byte Pair Encoding (BPE) configurations, specifically, merge operations on neural machine translation (NMT) performance for the Filipino-Cebuano language pair across various text domains. Results demonstrate that smaller BPE configurations, notably 2k, 5k, and 8k consistently yield higher BLEU scores, indicating improved translation quality through finer tokenization granularity. Conversely, larger BPE configurations and the absence of BPE result in lower BLEU scores, suggesting a decline in translation quality due to coarser tokenization. Additionally, these findings help us understand how the size of the model and how finely we break down words affect the quality of translations. This knowledge will be useful for improving translation systems, especially for languages that don’t have many parallel texts available for training.

## introduction a high-quality, large-scale parallel corpus is undeniably crucial for enhancing neural machine translation. unfortunately, parallel texts for many language pairs are scarce or nonexistent, limiting the effectiveness of many neural machine translation models. to improve translation for low-resource languages, several challenges must be addressed, including the scarcity of high-quality parallel data, the utilization of alternative sources such as monolingual resources and noisy comparable data, as well as parallel data in related languages, among other challenges. an alternative method for enhancing translation quality in low-resource settings involves incorporating linguistic features, such as lemmas, part-ofspeech (pos) tags, and dependency labels, into the nmt framework. this integration can aid in learning better token and sentence-level representations @xcite @xcite @xcite . however, the availability of these linguistic features and resources is also scarce or not readily accessible for many languages. subword segmentation is a fundamental preprocessing step for nmt and has been shown to significantly impact the quality of the final output @xcite @xcite @xcite . these methods enable nmt models to perform open-vocabulary translation by encoding rare and unknown words as sequences of subword units. this is particularly relevant for languages that form words through agglutination or compounding. although subword segmentation does not typically adhere to morphological constraints, it mimics these processes by learning the optimal segmentation from training data, thereby creating vocabularies of subword tokens capable of generating new words not seen during training. subword segmentation has become the de facto standard in neural machine translation @xcite . known subword segmentation algorithms are sentencepiece unigram lm @xcite , word-piece algorithm @xcite , and byte pair encoding (bpe) @xcite being the dominant approach @xcite . in this paper, we investigate the optimal configuration settings of bpe for translating from filipino to cebuano across various domains. we evaluated the results using intrinsic methods, such as the bleu score, and extrinsic methods, including an examination of the morphological and syntactic divergence of the translations. to the best of our knowledge, there has been no comprehensive experimentation and analysis like this conducted for the filipino-cebuano language pair or for any of the major austronesian languages. ## related work this study was initiated after reviewing the research conducted by @xcite . their systematic exploration of various numbers of bpe merge operations shed light on how these operations interact with model architecture, vocabulary construction strategies, and language pairs. their investigation aims to offer insights into selecting appropriate bpe configurations in future endeavors. their findings demonstrate that lstm-based architectures require experimentation with a wide range of bpe operations, as there is no single optimal configuration. conversely, transformer architectures tend to perform better with smaller bpe sizes. they emphasized the importance of careful consideration in selecting subword merge operations, as their experiments indicate that an ill-suited bpe configuration alone could lead to a decrease in system performance by 3-4 bleu points. several studies have delved into the vocabulary generated by bpe merge operations. @xcite 's work has investigated threshold vocabulary trimming in bpe subword tokenization-a postprocessing step that replaces rare subwords with their component subwords. their results suggest that while removing rare subwords is commonly recommended in machine translation implementations to reduce model size and enhance robustness, our experiments across a wide range of hyperparameter settings indicate that vocabulary trimming does not consistently improve performance. while the work of @xcite presented two straightforward randomized adaptations of byte pair encoding (bpe) and investigated their impact on a subsequent machine translation task. their study concentrates on translating into languages with complex morphology, aiming to determine whether the method of selecting subwords significantly influences performance. utilizing a bayesian linear model for analysis, they find that one variant performs nearly identically to standard bpe, while the other exhibits less performance degradation than expected. this suggests that while standard bpe is prevalent, there are intriguing alternative variations worth exploring. @xcite conducted detailed analyses across 47 typologically diverse languages and three parallel corpora, revealing that the types of recurrent patterns significantly influencing compression indicate morphological typology. for languages with richer inflectional morphology, early merges favor highly productive subwords, while languages with less inflectional morphology highlight more idiosyncratic subwords. both pattern types enhance compression efficiency. they emphasized that contrary to the belief that bpe subwords lack linguistic relevance, they found crosslinguistic patterns resembling those in traditional typology. ## experimental setup our experiments aim to understand the impact of adjusting the number of bpe merge operations across multiple domains in the filipino to cebuano neural machine translation (nmt). we evaluated the outcomes using bleu scores and by examining the actual translations. ## conclusions in this work, we have explored the efficacy of various bpe configurations on neural machine translation performance across multiple text domains for the filipino-cebuano language pair. the analysis demonstrated that smaller bpe configurations, particularly 2k and 5k, consistently yielded higher bleu scores across all domains, highlighting the importance of finer tokenization granularity in enhancing translation quality. conversely, larger bpe configurations and the absence of bpe resulted in lower bleu scores, indicating a loss of translation quality due to coarser tokenization. the performance variations across different domains underscore the domain-specific challenges in translation. for instance, the bible and news article domains exhibited the highest bleu scores at smaller bpe sizes, while the wikipedia domain presented more nuanced trends, with some larger configurations performing comparably to smaller ones. this indicates the complexity of domainspecific terminologies and structures in influencing nmt performance. the morphological and syntactic analysis revealed that the models' ability to handle affixation and other linguistic features varied with bpe size. smaller bpe configurations better preserved morphological integrity and syntactic accuracy, whereas larger configurations occasionally introduced errors, particularly in handling complex linguistic patterns. this underscores the critical role of appropriate bpe segmentation in maintaining linguistic fidelity in translations. furthermore, the study highlighted the significant impact of model size on scalability and deployment. larger models, associated with higher bpe configurations, pose challenges for distribution and real-time application due to increased computational and memory requirements. this necessitates a careful balance between tokenization granularity and model complexity to optimize performance while managing resource constraints. overall, the findings emphasize the importance of selecting optimal bpe configurations tailored to specific domains and linguistic features to enhance nmt performance. the study contributes valuable insights into the balance between tokenization granularity, model size, and translation quality, guiding future developments in both bilingual and multilingual nmt systems.