The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translation (2021)

Abstract: This paper evaluates the performance of several modern subword segmentation methods in a low-resource neural machine translation setting. We compare segmentations produced by applying BPE at the token or sentence level with morphologically-based segmentations from LMVR and MORSEL. We evaluate translation tasks between English and each of Nepali, Sinhala, and Kazakh, and predict that using morphologically-based segmentation methods would lead to better performance in this setting. However, comparing to BPE, we find that no consistent and reliable differences emerge between the segmentation methods. While morphologically-based methods outperform BPE in a few cases, what performs best tends to vary across tasks, and the performance of segmentation methods is often statistically indistinguishable.

## introduction despite the advances of neural machine translation (nmt), building effective translation systems for lower-resourced and morphologically rich languages remains a challenging process. the lack of large training data sets tends to lead to problems of vocabulary sparsity, a problem exacerbated by the combinatorial explosion of permissible surface forms commonly encountered when working with morphologically rich languages. current nmt systems typically operate at the level of subwords. most commonly, these systems achieve vocabulary reduction by decomposing tokens into character sequences constructed by maximizing an information-theoretic compression criterion. the most widely used subword segmentation method is byte pair encoding, originally invented in the data compression literature by @xcite , and introduced to the mt community by @xcite . another approach to open vocabulary nmt has been to compose characters or character n-grams to form word representations @xcite . as bpe has become mainstream, the question of whether segmenting words in a linguisticallyinformed fashion provides a benefit remains open. intuitively, the translation task may be easier when using subwords that contain maximal linguistic signal, as opposed to heuristically derived units based on data compression. the greatest benefit may come in low-resource settings, where the training data is small and biases toward morphological structure may lead to more reusable units. we seek to address this question by exploring the usefulness of linguistically-motivated subword segmentation methods in nmt, as measured against a bpe baseline. specifically, we investigate the effectiveness of morphology-based segmentation algorithms of @xcite and @xcite as alternatives to bpe at the word or sentence level and find that they do not lead to reliable improvements under our experimental conditions. we perform our evaluation using both bleu @xcite and chrf3 @xcite . in our low-resource nmt setting, all these methods provide comparable results. the contribution of this work is that it provides insights into the performance of these segmentation methods using a thorough experimental paradigm in a highly replicable environment. we evaluate without the many possible confounds related to back-translation and other processes used in stateof-the-art nmt systems, focusing on the performance of a straightforward transformer-based system. to analyze the performance differences between the various segmentation strategies, we utilize a bayesian linear model as well as nonparametric hypothesis tests. ## related work attempts to create unsupervised, morphologicallyaware segmentations have often been derived from the morfessor family of morphological segmentation tools @xcite . in addition to extensions of morfessor, such as cognate morfessor @xcite , @xcite and @xcite introduced the lmvr model, derived from morfessor flatcat @xcite , and applied it to nmt tasks on arabic, czech, german, italian, turkish and english, noting that lmvr outperforms a bpe baseline in chrf3 and bleu. contrary to their results, however, @xcite find that using lmvr yielded mixed results: on a kazakh-english translation task the authors observed marginal bleu improvements over bpe, whereas for english-kazakh, the authors reported lmvr to perform marginally worse than bpe in terms of chrf3. there have also been efforts to combine bpe with linguistically motivated approaches. for instance, @xcite propose to combine bpe with various linguistic heuristics such as prefix, suffix, and compound splitting. the authors work with english-german and german-english tasks, and observe performance improvements of approximately 0.5 bleu compared to a bpe-only baseline. as another example, weller-di marco and fraser (2020) combine bpe with a full morphological analysis on the source and target sides of an english-german translation task, and report performance improvements exceeding 1 bleu point over a bpe-only baseline. finally, even though @xcite originally only used the nmt training set to train their segmentation model, others have recently found benefit in adding monolingual data to the process. in particular, scherrer et al. (2020) used both sen-tencepiece and morfessor as segmentation models on an upper sorbian-german translation task and found a monotonic increase in bleu when the segmentation model was trained with additional data, while at the same time keeping the nmt training data constant. ## experiments to investigate the effect of subword segmentation algorithms on nmt performance, we train translation models using the transformer architecture of @xcite . we base our work on two recent datasets: flores @xcite , and select languages from the wmt 2019 shared task on news translation @xcite . corpus statistics for all corpora can be found in the flores dataset consists of two language pairs, english-nepali and english-sinhala. to add another lower-resourced language, we use the kazakh-english translation data from wmt19. in terms of morphological typology, both nepali and sinhala are agglutinative languages @xcite , as is kazakh @xcite . we conduct two sets of experiments on kazakh to investigate how the amount of training data influences our results: first, we train only on segmentation sentence original the nation slowly started being centralized and during sentencepiece the n ation sl ow ly start ed being cent ral ized and d ur ing subword-nmt the n@@ ation s@@ low@@ ly star@@ ted being cen@@ tr@@ ali@@ z@@ ed and d@@ ur@@ ing lmvr the nation s +low +ly st +ar +ted be +ing c +ent +ral +ized and d +ur +ing morsel the nation s@@ low +ly start +ed being cen@@ tr@@ ali@@ z +ed and du@@ r +ing the wikititles and news commentary corpora (train120k), followed by another set of experiments (train220k) where we include the web crawl corpus prepared by bagdat myrzakhmetov of nazarbayev university. we also conducted experiments with gujarati data from wmt19, but bleu scores were too low to allow for meaningful analysis. for our models, we generally follow the architecture and hyperparameter choices of the flores transformer baseline, except for setting clip norm to 0.1 and enabling fp16 training. despite the widespread use of auxiliary techniques such as back-translation we deliberately refrain from employing such techniques in this work. this is done to best isolate the effect of varying the subword segmentation algorithm, and to avoid the complexity of disentangling it from the effect of other factors. it should be noted, however, that such techniques were highly prevalent among of systems submitted to the kk↔en wmt19 news translation shared task: 64% used back-translation, 61% used ensembling, and 57% employed extensive corpus filtering @xcite . ## results and analysis our experimental results can be seen in overall, based on tables 3 and 4 , no segmentation method seems to emerge as the clear winner across translation tasks, although bpe applied at the token (subword-nmt) or sentence (sentence-piece) level performs well consistently. subword-nmt or sentencepiece perform best in 12 out of 16 cases (counting bleu and chrf3 for each translation task), while morphology-based methods rank best in 4 out of 16 cases. in particular, we note that morphology-based methods seem to achieve or tie the best bleu performance for translation tasks involving si, and best chrf3 performance for kk-en with smaller training data (train120k) as well as en-si. however, when using lmvr, we fail to find the significant gains in bleu compared to bpe reported by @xcite . comparing our results to @xcite , we note that the scores are similar, although not directly comparable as we report lowercased bleu ## conclusion and future work contrary to our hypothesis about the usefulness of morphology-aware segmentation, we see no consistent advantage, and possibly a small disadvantage, to using lmvr or morsel in this resource-constrained setting. by and large, our experiments and modeling show that no segmentation approach consistently achieves the best bleu/chrf3 across all translation tasks. bpe remains a good default segmentation strategy, but it is possible that lmvr, morsel, or similar systems may show larger performance advantages for languages with specific morphological structures. consequently, we believe further work is needed to better understand when morphology-aware methods are most effective and to develop methods that provide a consistent advantage over bpe. one such avenue of future work would be to broaden our analysis to more languages and include languages that are higher-resourced but morphologically rich and as well as ones that are lower-resourced but morphologically poor. @xcite , which we encountered during preparation of the final version of this paper, began to address these questions by comparing morfessor with bpe and their own bpe variant on finnish, quechua and spanish. an alternative approach which we intend to pur-sue in future work is experimenting with supervised morphological segmenters or analyzers that can be efficiently developed even in lower-resourced settings. incorporating such "gold standard" segmentations may make it clearer whether the unsupervised morphological segmenters are capturing linguistically-relevant structure. finally, there is the question of whether bpe can approximate a general representation for a language instead of converging on a corpus-specific set of subwords. to test this, one can add monolingual data and train the bpe segmentation on that larger data set. ideally the new, "enriched" segmentations would depend less on the specific vocabulary of the training corpus. as noted above, @xcite observed this approach to be helpful in terms of bleu. however, it remains unknown why the subwords derived from a larger corpus perform better, and whether better identification of morphological structure could be responsible. we hope that this work and these ideas will catalyze further research, and that efficient methods for translating to and from lower-resourced languages can be developed as a result. yves scherrer, stig-arne grönroos, and sami virpioja. 2020. the university of helsinki and aalto university submissions to the wmt 2020 news and low-resource translation tasks. in proceedings of the fifth conference on machine translation, pages 1129-1138, online. association for computational linguistics. rico sennrich, barry haddow, and alexandra birch. 2016. neural machine translation of rare words with subword units. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1715-1725, berlin, germany. association for computational linguistics. rico sennrich and biao zhang. 2019. revisiting lowresource neural machine translation: a case study. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 211-221, florence, italy. association for computational linguistics. antonio toral, lukas edman, galiya yeshmagambetova, and jennifer spenader. 2019. neural machine translation for english-kazakh with morphological segmentation and synthetic data. in proceedings of the fourth conference on machine translation (volume 2: shared task papers, day 1), pages 386-392, florence, italy. association for computational linguistics. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems, pages 5998-6008. sami virpioja, peter smit, stig-arne grönroos, mikko kurimo, et al. 2013. morfessor 2.0: python implementation and extensions for morfessor baseline. technical report. marion weller-di marco and alexander fraser. 2020. modeling word formation in english-german neural machine translation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4227-4232, online. association for computational linguistics.