Using Multiple Subwords to Improve E nglish- E speranto Automated Literary Translation Quality (2020)

Abstract: Building Machine Translation (MT) systems for low-resource languages remains challenging. For many language pairs, parallel data are not widely available, and in such cases MT models do not achieve results comparable to those seen with high-resource languages. When data are scarce, it is of paramount importance to make optimal use of the limited material available. To that end, in this paper we propose employing the same parallel sentences multiple times, only changing the way the words are split each time. For this purpose we use several Byte Pair Encoding models, with various merge operations used in their configuration. In our experiments, we use this technique to expand the available data and improve an MT system involving a low-resource language pair, namely English-Esperanto. As an additional contribution, we made available a set of English-Esperanto parallel data in the literary domain.

## introduction in this paper, we use the constructed language esperanto to illustrate potential improvements in the automatic translation of material from low-resource languages. languages are considered low-resource when there is little textual material available in the form of electronically stored corpora. they pose significant challenges in the field of machine translation (mt), since it is difficult to build models that perform adequately using small amounts of data. multiple techniques have been developed to improve mt in conditions of data scarcity. a popular approach is to translate indirectly via a pivot language @xcite @xcite . moreover, indirect translation can be used for creating additional training data. a further useful technique for expanding the dataset is backtranslation @xcite . this procedure consists of automatically translating a monolingual text from the target language into the selected source language, and then using the resulting parallel set as training data so the model benefits from this additional information. although the quality of these sentence pairs is not as high as that of human-translated sentences (the source side contains mistakes produced by the mt system), the pairs are still useful when used as training data, because they do often improve the models @xcite . nonetheless, for some languages, the available data are in such short supply that mt models used for generating back-translated sentences may produce a high proportion of noisy sentences. the use of noisy sentences for building mt models could ultimately have a negative impact on the quality of the mt system's outputs @xcite , and therefore they are often removed @xcite @xcite . we propose employing another technique to augment datasets: using the same set of sentences multiple times, but in slightly altered form each time. specifically, we modify the sentences by using different byte pair encoding (bpe) @xcite ) merge operations. we perform a fine-grained analysis, exploring the use of different splitting options on the source side, on the target side, and on both sides. ## previous work this research is inspired by techniques for augmenting the training set artificially. one of these techniques is back-translation @xcite , which involves creating artificial source-side sen-tences by translating a monolingual set in the target language. similar techniques include the use of several models to generate sentences @xcite , or the use of synthetic data on the target side @xcite . a technique that involves multiple segmentation is subword regularization @xcite , in which candidate sentences with different splits are sampled, either probabilistically or using a language model for training. in the work of @xcite , different splits are used to build an english-thai mt model. as the thai language does not use whitespace separation between words, different splits can be applied, to address the fact that all the words and sub-words are joined together in the final output. more recently, @xcite introduced bpe-dropout, an improvement on standard bpe consisting of randomly dropping merges when training the model, such that a single word can have several segmentations. ## the esperanto language this article is concerned with improving mt models for esperanto, the most successful constructed international language @xcite . it was created in the late nineteenth century, and is said to be currently spoken by over 2 million people, spread across more than 100 countries @xcite . during its first century of development, esperanto was principally maintained by means of membership-based organisations. currently, internet applications such as duolingo are supporting the wider spread of the language among new enthusiasts. while many esperanto speakers have sought to develop the language through translation, the body of work available -particularly in digital formats -remains relatively small, making esperanto a clear example of a low-resource language. esperanto loosely derives its lexicon from several indo-european languages, and shares some typological characteristics with, among others, russian, english, and french @xcite . in contrast to most natural languages, esperanto's most distinctive characteristic is its regularity. the grammar consists of a very limited set of operations, to which there are, in principle, no exceptions. furthermore, the language is agglutinative, and its suffixes are independently meaningful and invariable. for instance, virino, the word for "woman", con-sists of the compound parts vir [adult human], in [female], and o [entity] (as the 'o' ending is used for all nouns). the word for "mother", patrino, largely refers to the same semantic categories, and is therefore structurally highly similar. as a consequence of this internal consistency, esperanto learners can quickly expand their vocabulary by learning to segment words into their various parts, which can then be used to construct new words by morphological analogy. because of its affinity with many other languages, and because of the thoroughly logical composition of its vocabulary, esperanto has historically been central to several experiments in mt, most notably regarding its potential function as a pivot language between european languages @xcite . in this study, however, we focus on automatic translation into esperanto for its own sake. ## research questions we propose building mt models using training data composed of a dataset split into multiple variants with a different configuration of bpe, as presented in figure 1 . at the top of the figure, one can see that the same parallel set has been processed using bpe with 89,500, 50,000 and 10,000 operations (trained separately for each language). the mt model represented on the left has been built using the same dataset replicated three times, the only difference being that on the target side, different splits were implemented. similarly, the mt model in the centre is built with different splits on the source side. the last model, represented on the right, combines different splits both on the source and the target side. in order to evaluate the models, we use a test set that is split with a single bpe strategy (i.e. using 89,500 merge operations, the default proposed in the work of @xcite ). therefore, using different merge operations on the source side of the training data may not have as big an impact as when they are applied to the target side (not all the words will match those in the test set). however, the addition of other bpe configurations could in principle still be useful to improve modeling for the source language. in section 5 we describe the settings of the mt and the data used for training. in section 6 we analyze the results achieved by the baseline system. this paper's experiments are divided into three sections. each of these sections describes and also provides the evaluation of a model. the sections are the following: in section 8, we compare translation examples from the different models and analyze the different outcomes. finally, in section 9 we conclude and propose how these experiments could be expanded in future work. ## experimental settings the nmt systems we build are transformer @xcite models, based on opennmt @xcite . models are trained for a maximum of 30k steps using the recommended parameters. 1 we have selected the model with the lowest perplexity on the development set. ## baseline mt in the rows of the table correspond to the evaluation of the model, using the same data. the only difference is the number of bpe merge operations that have been used on the target side. as the test set is split using 89,500 merge operations, it would not be beneficial to apply bpe with merge operations other than 89,500 on the source side. in fact, when using bpe with 50,000, 20,000 and 10,000 operations on the source side, the bleu score for the translation of the books data is only 5.75, 5.70 and 5.76, respectively, and 14.30, 13.24, and 14.53 for salome ## comparison of outputs in the first example, drawn from alice in wonderland, contains a joke. alice, who is collecting her thoughts, aims to voice her opinion, and starts out by saying i don't think.... before she can finish her sentence, however, the mad hatter interrupts her by stating that in that case, she should not speak. the human esperanto translation makes this joke very explicit by repeating the emphasis on 'not thinking', whereas in english the transition is more subtle. two of the systems, while differing in exact word order, succeed in reproducing the joke (trg89500 and trg89500 & trg10000). in the other two models, either the crucial element do [so], which realises the inference, is omitted, or the meaning is mistakenly changed to a positive imperative: vi devus diri [you should say]. it can further be observed in the sentences that none of the systems translates the hatter's name meaningfully. either the name remains the same, or it is slightly altered from the original, in a seemingly random manner. interestingly, alice's name is adapted to alico, which conforms to the rule that all esperanto names end in -o (or, in some cases -a), but the adaptation does not equal the human choice for alicio. the second example, also taken from alice's adventures in wonderland, is concerned with a particular fixed expression in the english language: venture to say. the baseline system does not translate this mark of politeness, while the other models do provide varying translations (i.e. decidis, sukcesis and entrepenis, which correspond to the past tenses of the verb to decide, to succeed and to undertake). while none of them is completely correct (when compared to the human translation), all of them are fairly transparent in context, and foreground different aspects of meaning contained in the english venture. with reference to the salome test set, we find in the entire translated text numerous small and relatively inconsequential vocabulary differences across systems (e.g. veston or mantelon for referring to a piece of clothing), as well as varying preferences for orthographically similar verb tenses (e.g lacigis or lacigas, past and present tense of the verb to tire or wear out). at times, the systems differ in their translation of multi-word units such as sacred person, which is translated either as the literal sankta homo or as the more interpretative sanktulo @xcite . overall, the systems perform well when translating the play's dense symbolism, as illustrated in the examples in the table are similes, which start with the explicit comparative phrase it is like. in the first example, the baseline system does not manage to reproduce the reference to serpentoj [snakes], although the mention of turmentoj [afflictions] does offer an interesting metaphorical perspective. the system src89500 & src10000 does not produce a correct translation, but those systems trained with different splits on the target side (i.e. the src10000 & src89500 and all systems) provide a remarkably good translation of the source. similarly, in the last example included in the table, the baseline system fails to reproduce the meaning of the original (the knife falls apart instead of cutting the fruit), whereas all systems with multiple segmentation are successful in conveying a variant of the poetic image presented in the source text. in short, the examples in ## conclusion and future work in this work, we have aimed to improve an english-esperanto mt system by using multiple instances of the same sentence pair, split with different configurations of bpe. in our experiments, the best performance tends to be achieved when splitting strategies are applied both on the source and target side, duplicate parallel sentences are removed, and the number of merge operations used are very different from each other. in our experiments, the best results are achieved when all the split-combinations are used on both sides. although the goal of these experiments is to find a technique to improve the mt models when the available data are very limited, this technique could also be applied in scenarios where data are abundant. it should be noted that esperanto is perhaps a particularly suitable candidate for word-split methods, as the language's vocabulary consists of fixed chunks that are combined to form transparent compounds. however, the techniques applied here are in principle language-independent. finally, although we demonstrated that combining sentences with different merge operations improves the model, in this paper we could not determine the best configuration to use. similarly, the test set that we used was processed using 89,500 merge operations. if the test set had been processed with a different bpe configuration the performance could have been different, especially when using models with different split configurations on the source side. extensions of this work could involve finding an optimal configuration for achieving the best results, or testing the performance when combined with other word-splitting techniques.