Love Thy Neighbor: Combining Two Neighboring Low-Resource Languages for Translation (2021)

Abstract: Low-resource languages sometimes take on similar morphological and syntactic characteristics due to their geographic nearness and shared history. Two low-resource neighboring languages found in Peru, Quechua and Ashaninka, can be considered, at first glance, two languages that are morphologically similar. In order to translate the two languages, various approaches have been taken. For Quechua, neural machine transfer-learning has been used along with byte-pair encoding. For Ashaninka, the language of the two with fewer resources, a finite-state transducer is used to transform Ashaninka texts and its dialects for machine translation use. We evaluate and compare two approaches by attempting to use newly-formed Ashaninka corpora for neural machine translation. Our experiments show that combining the two neighboring languages, while similar in morphology, word sharing, and geographical location, improves Ashaninka– Spanish translation but degrades Quechua–Spanish translations.

## introduction low-resource languages (lrl) can be defined as languages that suffer from the presences of insufficient parallel source-target data. until recently, in order to translate lrls, rule-based (rbmt) or statistical-based machine translation (smt) systems have been used with a combination of features and heuristic approaches to create a model that could predict target-side translations based on probability techniques given a source sentence (also known as a segment). with the rebirth of neural machine translation (nmt) in recent years thanks to higher-compute system availability, neural approaches have been used to jointly learn from several source and target segments @xcite @xcite avoiding the highly laborious process of creating rules and features to translate using previous rbmt and smt systems. the majority of research that uses nmt for lrls tends to show how the combining of two or more source-side languages to one target-side language can help translate low-resource languages by imputing word-level features from a higher-resource language to a lower-resource language. one such case @xcite translates quechua, a peruvian lrl, to spanish, an hrl, using finnish, another hrl, in an approach called bpe-guided based on glossary def-initions found from suffixes on wikipedia foot_0 . their work which uses an nmt system based on byte-pair encoding (bpe) @xcite with a long short term memory (lstm) @xcite was found to outperform other systems measured according to human and system evaluations using bleu @xcite . research has also been performed by @xcite on a neighboring peruvian language called ashaninka. ashaninka has less resources than quechua and is spoken by fewer people. there are nearly 70,000 native ashaninka speakers @xcite as compared to around 5 million native quechua speakers foot_1 and both languages can be broken down into different dialects. the amount of resources available for ashaninka is on the order of 8,000 sentences (or segments) whereas quechua data is about 40,000 segments and growing. @xcite dedicated their initial work on ashaninka to language normalization by creating a finite-state transducer based on previous quechua work @xcite . they left for future work the inclusion of ashaninka in an nmt system. in order to advance the work by @xcite , we use resources from their published articles available online foot_2 to extend their experiments which, in turn, marks the first time, to our knowledge, that an ashaninka-spanish machine translation (mt) system is introduced to the mt research community. our hope is that, since finnish and quechua were found to be successful in previous work @xcite due to their highly-similar morphology, the addition of ashaninka as source-side input should increase performance since quechua and ashaninka are from the same region, display similar morphological constructs, and even share loaned vocabulary words where higher-resource languages (quechua and spanish) are found in the lower-resource language (ashaninka). our effort is a three-fold, novel, experimental introduction for the two peruvian languages as seen below: in order to realize the three points, we narrate the following. first, in section 2, we describe related approaches not mentioned in section 1. next, we analyze quechua and ashaninka similarities and differences in section 3. our methodology and approach are detailed in section 4 along with the experimental settings in section 5. we then provide results in section 6 that show how combining quechua and ashaninka together perform on both a quechua and ashaninka test set. lastly, we conclude with an explanation on our findings and potential future research lines in section 7. tantug and adalı (2018) focus on agglutinating languages by using eight informal targetside, rule-based, edits. their work can be considered similar to the work from @xcite due to the way it handles morphology and knowledge transfer. however, they use discrete rules meant to work with a statistical disambiguation system for combining the source and target language. our aim is to show that nmt could be used to learn similar rules without human intervention. nonetheless, we feel that their work could be included for comparison in future iterations. @xcite use a neural machine translation system to first learn aligned words that form an encoded vector and then translate them. this work is similar to ours in its approach; however, our work is for an extremely low-resource language (ashaninka) and depends on character-level differences not performed in their work. we mirror @xcite 's approach by using the "opennmt-lstm" system mentioned in @xcite . @xcite 's results show an increase of 5 bleu when combining languages; our results are similar when using quechua as the high-resource language. other work @xcite tend to focus on the addition of several languages with high resources as was done by @xcite with the inclusion of finnish, a high-resource language. in this case, we are adding the lower-resource language, ashaninka, with hopes to better the higher-resource language, quechua. additionally, other work @xcite points out that bilingual nmt models may require adjustments when multilingual models perform better. their work is considered helpful; but, at this early stage of investigation, we lean on the work from @xcite for guidance. ## morphology quechua and ashaninka are morphologically similar at first glance. however, the deeper differences explained here help to understand the results presented in section 6. in this section, we provide an in-depth analysis of both languages based on previous work @xcite . the comparative analysis of the two language's grammatical makeup and morphology, to our knowledge, has not been taken into account by other research, specifically for machine translation. like many native north and south american languages, ashaninka and quechua are both polysynthetic and agglutinating @xcite , they add prefixes or suffixes to a root morpheme which expand or change a word's meaning. an example follows of the two languages agglutinating similarity. verbal conjugation generally inflects and agglutinates in both languages. in quechua, verbs use suffixes to express the present, past, or future tense. on the other hand, in ashaninka, most verbs do not take tense inflection into account, instead they use a category called the "reality status" which distinguishes between two types of events: (1) past and present (real) events or (2) future (unreal) events. @xcite "to come" quechua ashaninka conjugation hamu-ni no-pok-i "i come" hamu-rqa-ni no-pok-i "i came" hamu-saq no-m-pok-e "i will come" above, we see how the verb "to come" is conjugated for quechua and ashaninka. there is a clear distinction between present (hamuni), past (hamurqani), and future (hamusaq) tenses for the root quechua morpheme hamu. contrastingly, we see how ashaninka uses the real/unreal method described, present and past (nopoki) are the same but the future (nompoke) is different for the root ashaninka morpheme pok. other linguistic differences also exist with respect to suffixes and their order. more specifically, the phrasal order differs such that quechua usually takes a subordinate clause proceeded by the verb while ashaninka is the opposite. additionally, the three languages (quechua, ashaninka, and spanish) contain words in written texts that can be considered unknown, or loaned, words that are inherited from their higher-resource language where quechua inherits from spanish and ashaninka inherits from both quechua and spanish. the overlapping words and other differences mentioned are found in the corpora from the work mentioned @xcite which contains normalized texts from corpora created in the past @xcite ; @xcite . ## methodology from the description in section 3, it is clear that, while initially similar, the morphological makeup of ashaninka is different than quechua. our experiments determine if it is better to use ashaninka or finnish as a language for transliteration in a nmt system when translating quechua and ashaninka to spanish. the inclusion of finnish as a source language in both quechua and ashaninka translations to spanish is motivated by @xcite which showed that neural machine translation was better when including finnish as a source language during training. our experiments are based on previous work @xcite which experiments with quechua 4 , finnish 4 , and ashaninka 5 as the source languages and spanish as the target language. we use their translation and normalization approaches to compare the two neighboring language's (quechua and ashaninka) translations into spanish using the nmt system described below. the best performing system from @xcite 's work is a nmt system first used 4 quechua and finnish are the source languages in @xcite ). 5 ashaninka was not translated into another language in @xcite . for development (called opennmt-lstm) and later in testing (called opennmt). 6 we compare its performance by using quechua, finnish, and ashaninka to train the nmt system in various combinations (see train languages in the results show the performance of the nmt system when using ashaninka, a neighboring language (about 10 km away), and finnish, a language that is of high geographic distance (about 8,000 km away), as source languages for translating quechua to spanish. additionally, experiments are performed to show how well quechua and finnish perform as source languages when translating ashaninka to spanish. the implication is that since finnish is agglutinative and polysynthetic and it has been shown to improve performance when translating quechua to spanish @xcite , it should help when translating from both quechua and ashaninka to spanish. the next section describes experimental settings for all languages. ## experimental settings our experiments mirror previous experiments @xcite in terms of the corpora and nmt system used. since we combine languages from both works, some of the corpora and languages used as nmt system input is different. in this section, we present those input changes and reiterate the similarities to previous work. first, for ashaninka text to be used as input into the nmt system, we transform it using the ashmorph @xcite normalization technique. for purposes of ashaninka inclusion in the experiments, there are 521 ashaninka training sentences (or segments), 111 development segments, and 111 test segments. they are used in three different training experiments: (1) quechua+finnish+ashaninka, (2) quechua+ashaninka and (3) ashaninka only; and, in one development and test direction (ashaninka-spanish). all of the corpora is randomly selected from the developoment corpora (cushimariano romano and sebastián q., 2008) used previously @xcite . second, for quechua normalization as input to the nmt system, a morphological normalizer @xcite ) from previous work @xcite is used. quechua is used as a training language in all of our training experiments except for when ashaninka is tested in isolation. the quechua corpora consists of 17,500 training segments, 2,500 development segments, and 5585 test segments all randomly chosen from @xcite 's experiments originated from the opus corpus foot_6 @xcite and used in three different training settings: (1) quechua+finnish, (2) quechua+finnish+ashaninka, and (3) quechua+ashaninka; and, in one development and test direction (quechua-spanish). third, finnish and spanish, both considered high-resource languages, are more plentiful. like the work from @xcite , we use the jw300 corpus @xcite . since spanish is the target language in all cases, finnish is the only high-resource language included for training. we use 149,251 finnish segments for training in two systems: (1) quechua+finnish and (2) quechua+finnish+ashaninka. spanish is used only for parallel development for testing with quechua-spanish and ashaninka-spanish language pairs. all segments for all languages were tokenized and true-cased using moses @xcite after normalization. to summarize our validation technique for the neural mt system experiments, we use two source-target pairs: quechua-spanish and ashaninka-spanish. for example, for the qu+fi+cni system in the nmt system used for all experiments is the system described in @xcite 's dev phase called opennmt-lstm. the system is trained for 100,000 epochs and it is a 2-layer lstm model @xcite with 500 hidden units, dropout of 0.3, and uses stochastic gradient descent as the learning optimizer along with a batch size of 64. to evaluate the nmt system, we use bleu @xcite like the work from @xcite . the next section explains how previous work @xcite was used to test the neighboring quechua and ashaninka languages with the nmt system proposed. ## results the experiments in our results are aligned with what has been discussed in section 3 section at a high level -ashaninka and quechua appear similar in linguistic nature at first glance; however, at a deeper evaluation, the lack of resources and complex grammatical differences decrease qu-es translation performance. on the other hand, similar to work from @xcite , we have shown that by adding quechua resources to ashaninka, there is a gain of 4.6 bleu. in all other cases where ashaninka was combined with quechua or finnish, the performance degraded for qu-es translations and only very slightly (.2 bleu) increased in one cni-es case. 8 another interesting takeaway is that finnish remains the better language to combine with quechua when translating quechua to spanish. this is due to the large amount of finnish training examples (149,251) compared to the small amount of ashaninka training examples (521). in actuality, the bleu score of the qu+cni trained system is the same as the bleu score of using qu+es alone in training reported by @xcite . this leads us to believe that if there were more ashaninka training examples the potential to outperform finnish as the transfer-learning language is high. ## conclusion and future work we have shown that while previous work combining languages may seem viable for lowresource languages, in some cases, while languages seem similar at first glance, results may differ. this is clear from our experiments with quechua and ashaninka that show performance loss when adding them together for transfer-based learning in an nmt system. nonetheless, it would be advantageous to try other techniques such as back-translation @xcite @xcite to create more synthetic ashaninka data since, at this point, finnish provides more gain when combined with quechua than ashaninka does. future lines of investigation will include a supervised version of the ashmorph @xcite algorithm with the intent to automate sub-segment level selection. the plan is to improve ashaninka to spanish translations by first creating more human-evaluated training data and, second, experimenting with several other resources to create more synthetic data. experimentation should also explore other similar languages since quechua seems to help (not hurt) ashaninka to spanish translations.