Morphologically-Guided Segmentation For Translation of Agglutinative Low-Resource Languages (2021)

Abstract: Neural Machine Translation (NMT) for Low Resource Languages (LRL) is often limited by the lack of available training data, making it necessary to explore additional techniques to improve translation quality. We propose the use of the Prefix-Root-Postfix-Encoding (PRPE) subword segmentation algorithm to improve translation quality for LRLs, using two agglutinative languages as case studies: Quechua and Indonesian. During the course of our experiments, we reintroduce a parallel corpus for Quechua-Spanish translation that was previously unavailable for NMT. Our experiments show the importance of appropriate subword segmentation, which can go as far as improving translation quality over systems trained on much larger quantities of data. We show this by achieving state-of-the-art results for both languages, obtaining higher BLEU scores than large pre-trained models with much smaller amounts of data.

## introduction subword segmentation is a common technique used to improve machine translation quality due to its ability to reduce the vocabulary size of input text. unsupervised techniques such as byte-pair encoding (bpe) @xcite are prevalent in most nlp tasks. on the other side of the spectrum, state-of-the-art morphological segmentation is achieved using dedicated neural seq2seq models @xcite . neither of these however, are well-suited for low-resource languages (lrls). bpe was found to oversplit roots of infrequent words in both english and japanese @xcite . lower bleu scores in quechua-spanish models segmented by bpe @xcite suggest similar side-effects for quechua. neural morphological segmentation models require large amounts of morpheme-labeled training data, which often does not exist at all for lrls. we propose the use of the prefix-root-postfix-encoding (prpe) algorithm @xcite as an alternative for subword segmentation. prpe is able to draw upon linguistic knowledge without needing large amounts of labeled training data, making it a middle-ground between bpe and neural seq2seq that is ideal for lrls. prpe is a semi-supervised word segmentation algorithm that uses subword statistics to identify and learn the prefixes, roots, and postfixes of words in a corpus @xcite , and can be guided using a language-specific heuristic. using the generated lists of roots and affixes, the algorithm performs subword segmentation that only appears to be morphologically grounded; prpe does not use any actual linguistic/morphological rules. this makes it wellsuited for studying lrls, as it only requires a surface level of understanding to tune the heuristic for a language instead of dedicated linguistic rules or large amounts of labeled training data. we experiment with two distinct agglutinative lrls, indonesian and quechua, as we hypothesized that prpe would naturally work well in the morpheme-heavy environment of these languages. this is because words in agglutinative languages are constructed via a series of affixes, leading to large amounts of information expressed in a single word due to the presence of many morphemes. as such, machine translation for these languages is particularly challenging due to the increased vocabulary size and more frequent appearance of rare words @xcite . quechua in particular is highly agglutinative; multiple suffixes are appended to modify a root to denote tense, mood, person, and number @xcite . to investigate the effectiveness of prpe in improving machine translation quality, we conduct experiments using two distinct language pairs, quechua to spanish and indonesian to english, across multiple domains of corpora. we accomplish this by training lstm @xcite and transformer @xcite models on text segmented by prpe and compare those with models trained on other segmentation methods. our experiments show that prpe subword segmentation can lead to significant improvements in machine translation performance, outperforming prior benchmarks with models pre-trained using masked language modeling @xcite , transfer learning @xcite , and models trained on much larger datasets @xcite . ## related work the current segmentation standard for most nmt systems is the unsupervised method of byte-pair encoding @xcite . bpe initially represents the corpus at a character level, after which pairs of the most frequently occurring symbols are iteratively merged together to form the vocabulary. however, recent works have shown that a unigram language model for segmentation @xcite , another unsupervised method, appears to be the better alternative. @xcite found unigram models better preserved roots and split affixes compared to bpe in english and japanese. @xcite observed similar benefits for two lrls: swahili and turkish. @xcite introduced the prpe algorithm by experimenting with an english-latvian pairing in both translation directions. prpe utilizes a proposed 'root alignment principle' -collecting statistics about prefixes and suffixes before aligning roots with the most frequent prefix and suffix. aside from the differences in languages used, our work also differs from @xcite in terms of how the algorithm is incorporated into the overall segmentation pipeline. they also did not consider running prpe standalone, all text segmented with prpe in their experiments were post-processed with bpe. in addition to this method, we also explore segmentation results obtained solely with prpe, as well as those obtained from multiple iterations of prpe. there have been several studies on nlp for quechua. rios (2016) created a language toolkit for quechua translation, which included a text normalizer, spell-checking, and morphological analyzer. @xcite focused primarily on translation from quechua to spanish. they proposed bpe-guided, a method to guide the bpe segmentation algorithm for quechua by feeding bpe a dictionary of words to ignore during segmentation. they utilized transfer learning from finnish (a high-resource agglutinative language) to obtain substantial improvements in bleu. it was also in this study that @xcite first suggested the use of prpe for quechua translation. @xcite conducted research on multilingual translation for four peruvian languages paired with spanish: aymara, ashaninka, quechua and shipibo-konibo. pre-processing was done using the unigram model @xcite ) trained across a multilingual corpus. quechua, the language with the most resources among the four, suffered in performance when trained on the multilingual task rather than solely quechua and spanish. quechua was also recently featured as part of the americasnlp shared task @xcite , where participants were asked to translate spanish text to quechua among other indigenous american languages. compared to quechua, indonesian has enjoyed much more attention in nlp research. extensive work has been done on computational approaches in indonesian morphological analysis, such as morphind @xcite and later on malindo morph @xcite , both of which created morphological dictionaries and supervised morphological analyzers for the language. @xcite conducted extensive benchmarks of the current state of indonesian-english nmt, evaluating performance across a multitude of domains such as news, religious text, and conversational text. @xcite extended their work to improve translation for colloquial indonesian to english by constructing a synthetic training corpus machine-translated from formal indonesian. ## prpe as suggested by its name, prefix-root-postfix-encoding (prpe) separates a word into three main parts, a prefix, a root, and a postfix. postfixes can be further broken down into a suffix and an ending. instead of the character pairs of bpe, left and right substrings are used for segmentation. left substrings are considered as potential prefixes, while right substrings are considered potential postfixes. similar to bpe, prpe is also split into a learning phase and application phase, the former of which @xcite outlines as four main steps: the learning phase of prpe can also take in a heuristic to help it determine whether a subword unit is a good candidate for a certain affix type. in other words, a set of hyperparameters that determine the threshold for affix candidacy. for example, an english heuristic could help identify statistically probable prefixes by using a list of known linguistic prefixes (such as preor non-) and some maximum character length. a left substring would be considered a potential prefix if it is found in the list or is within the maximum character length. we apply the same principle when creating the quechua and indonesian heuristics by using a list of common prefixes and suffixes, obtained from @xcite , @xcite , @xcite for quechua and indodic 1 for indonesian. the exact heuristic implementations can be found online 2 . this approach is cross-dimensional, and thus makes the algorithm easily extendable to other languages due to the large amount of affix information publicly available. this learning phase generates 6 files that are used during the application phase: ranked lists of prefixes, roots, postfixes, suffixes, endings, and words that the algorithm has learned. a word is segmented by generating all possible segmentations and choosing the highest ranked candidate. it is important to reiterate that prpe is not attempting true morphological segmentation of a word into its linguistic morphemes, but rather into subword units that it deems statistically likely to be prefixes, roots, postfixes, suffixes, and endings. this constitutes the "morphologically guided" portion of prpe, as it allows for sub-word tokenization that resembles morphologically motivated segmentation. ## datasets the main dataset used for analyzing performance was the jw300 texts @xcite from the opus corpus @xcite , comprised of jehovah's witness scripture across a variety of languages. despite the dataset's domain specific content, it is also one of the largest parallel texts publicly available for quechua. for the quechua-spanish pair, we used the version of the dataset made publicly available by @xcite , already split into 17,500 training sentences, 2,500 validation sentences, and 5,585 test sentences. for indonesian-english we use the an altered version of jw300 provided by @xcite , which also includes bible and quran texts gathered from bible-uedin and tanzil respectively @xcite . this dataset is split into 579,544 training sentences, 5,000 validation sentences, 4,823 test sentences. we denote both of these as the religious datasets. to ascertain the effects of prpe outside of the religious domain, we also conduct experiments using general language corpora. we include two different general language datasets for quechua-spanish. the first is comprised of financial news articles from dw news, originally created by @xcite for statistical machine translation. however, it was only available as parallel parse trees in an xml format, rendering it unusable for nmt models trained on parallel plaintext. we manually align and clean the raw source text data, filtering out uncertain alignments. the entire cleaned 2,018 line corpus is denoted as the financial dataset. the second corpus used was the 100 sentence magazine dataset created by @xcite . both general-language quechua-spanish datasets were used solely for testing due to their small size. for the indonesian-english pair, despite having access to much larger general language datasets, we chose to use the low-resource news dataset created by @xcite . this is because of the relatively large size of the religious indonesian-english dataset compared to many truly low-resource languages (although it is still substantially smaller than most highresource language datasets); we wanted to examine the benefits of prpe in a very low-resource setting for both language pairs. the news dataset is split into 38,469 training sentences, 1,953 validation sentences, and 1,954 test sentences. ## experimental setup we separate our experiments into two stages: development and testing. we use the development stage to experiment with different model architectures and parameter settings, the best performing of which were carried over to the testing stage. in the testing stage, we evaluate the models on both in-domain and out-of-domain corpora after they were trained on a specific dataset. our pipeline includes two pre-processing steps: tokenization and subword segmentation. tokenization is done using moses tokenizer @xcite , demonstrated by @xcite to be effective in translation tasks. segmentation is done solely on the source language (either indonesian or quechua) text, using one of the methods described above: prpe, multi-prpe, and prpe+bpe. the target language is left unsegmented. to establish a baseline for comparison, we also conduct experiments with unsegmented text and text segmented with bpe and a unigram language model. we use the default sentencepiece vocabulary size of 8000 for all segmentation methods. we use the sacrebleu @xcite implementation of bleu @xcite as our primary metric for evaluation of translation quality to allow for a comparison with other studies. @xcite used bleu to evaluate their nmt systems and found it to be correlated with human judgement for the quechua-spanish translation direction. @xcite used bleu scores from sacrebleu to benchmark indonesian-english translation. ## results bleu scores from the development stage ( perhaps most interestingly, the models trained on the smaller two training sets (the quechua-spanish religious set and the indonesian-english news set) consistently yielded bet-ter performance when trained on the lstm architecture compared to the transformer no matter the segmentation method used, with differences as high as 2.6 bleu. @xcite also observed the same pattern in their experiments with quechua-spanish translation. these results may suggest that the best performing architectures and techniques in high-resource settings may not be transferable to low-resource translation. we leave further evaluation to future studies. as such, the lstm models trained on the quechua-spanish religious set and the indonesian-english news set, instead of their transformer equivalents, were carried over for testing. in-domain testing results for quechua-spanish were consistent with the development stage with prpe outperforming other segmentation methods ( prpe performed well during both in-domain and out-of-domain testing for the indonesian-english pair . in-domain results for the religious dataset ( encouraged by the results on the indonesian-english religious set, we set up additional experiments using prpe in an effort to match the google translate benchmarks obtained by @xcite on the validation set (test set scores were not available), which obtained a bleu score of 29.1 ( ## limitations while we show that prpe was able to bring substantial gains in translation quality, there are still constraints that limit its applicability. the most immediate is the pre-requisite for some amount of linguistic knowledge (a list of affixes) during the construction of its heuristics due to its semi-supervised nature. an extension of this limitation is that heuristics are thus language specific, making it less applicable in cross-lingual scenarios (although a multilingual heuristic could be developed to alleviate this problem). finally, the effectiveness of the algorithm on non-agglutinative languages is unclear. while @xcite showed that prpe brought incremental improvements in bleu score for the non-agglutinative english-latvian pairing, their experiments also had text further segmented with bpe (which was shown in our results to decrease the benefits of prpe for agglutinative languages). ## conclusion we introduced the use of the prpe algorithm for morphologically-guided subword segmentation and evaluate it on two distinct low-resource, agglutinative languages: quechua and indonesian. during the development of our experiments, we reintroduced datasets previously unavailable in parallel plaintext for nmt by manually re-aligning raw source data. we found that subword segmentation can have an especially large impact on low-resource translation; unsuitable segmentation methods can actually lower bleu score when compared to unsegmented text, while effective segmentation can produce moderate to large gains. our results show that segmentation using prpe can lead to significant improvements in translation quality when evaluated via bleu score, out-performing pre-trained, higher-resource models, making the algorithm ideal for low-resource languages that lack the large amounts of training data often necessary for neural machine translation.