Improving Low-Resource Machine Translation for Formosan Languages Using Bilingual Lexical Resources (2024)

Abstract: This paper investigates how machine translation for low-resource languages can be improved by incorporating information from bilingual lexicons during the training process for mainly translation between Mandarin and Formosan languages, which are all moribund or critically endangered, and we also show that our techniques work for translation between Spanish and Nahuatl, a language pair consisting of languages from completely different language families. About 70% of the approximately 7,000 languages of the world have data in the form of lexicons, a valuable resource for improving low-resource language translation. We collect a dataset of parallel data and bilingual lexicons between Mandarin and 16 different Formosan languages and examine mainly three different approaches: (1) simply using lexical data as additional parallel data, (2) generating pseudo-parallel sentence data to use during training by replacing words in the original parallel sentence data using the lexicon, and (3) a combination of (1) and (2). All three approaches give us gains in both Bleu scores and chrF scores, and we found that (3) provided the most gains, followed by (1) and then (2), which we observed for both translation between Mandarin and the Formosan languages and Spanish-Nahuatl. With technique (3), we saw an average increase of 5.55 in Bleu scores and 10.33 in chrF scores.

## introduction evidence shows that neural machine translation (nmt) systems do not work well with languages that lack parallel data @xcite , and typically, such machine translation systems require millions of parallel sentences as training data, which are only available for a limited number of language pairs @xcite . however, there are several low-resource languages with a significant number of speakers, such as burmese and @xcite , highlighting the need for more attention to be paid to the development of machine translation systems for these languages. for example, while there are nearly 280 million english-french parallel sentences in opus @xcite , there are only about 700,000 english-burmese parallel sentences, even though burmese (myanmar) has tens of millions of speakers @xcite . furthermore, it is important to improve machine translation for all languages, even those with few speakers, to help provide fair access to technology. though parallel data are only available for a limited number of language pairs, bilingual lexicons are an often overlooked resource that are more readily available and less expensive to produce @xcite . language documentation efforts typically produce bilingual lexicons first, and about 70% of the approximately 7,000 languages of the world have such lexicons @xcite . they may be advantageous in that they can provide information about infrequent terms that may not appear in parallel data, but they also are not as helpful as parallel data in that they do not give information about how a word may be translated differently depending on the context @xcite . however, despite the lack of contextual information, bilingual lexicons still provide additional valuable information that machine translation systems can use. this paper explores how bilingual lexicons can be used as a complement to parallel data in the context of low-resource machine translation for asian languages. in contrast to prior work which so far has mostly used replacement-based techniques which leverage bilingual lexicons by generating pseudo-parallel data, in this paper we propose to simply use lexical data as additional parallel data for training. to test our idea, we introduce a dataset containing parallel sentences and bilingual lexicons for 16 formosan languages, which have a great deal of diversity in linguistic struc-ture, which we use to propose an extensive empirical framework. concretely, we aim to answer the following research questions: (rq1) can adding lexical information as parallel data improve translation quality?, (rq2) how much lexical information is needed to improve translation quality?, (rq3) how does using lexical information compare with using parallel data?, (rq4) how does using lexical information compare with pseudoparallel data? our results demonstrate that adding lexical information during the training process can produce significant improvements in translation quality for real-world, low-resource language pairs, using formosan languages and mandarin as an example. we also examine the relationship between improvements in translation quality and the amount of additional lexical information provided, and compare the effect of using lexical information with the effect of using additional parallel sentence data. additionally, we also demonstrate that our methods improve translation quality for one non-asian language pair, spanish-nahuatl, suggesting that our technique may generalize to other language pairs. ## related work bilingual lexicons have proven useful in aiding a large a variety of downstream tasks such as semantic role labeling @xcite , sentence alignment @xcite , and partof-speech tagging @xcite . @xcite explored the use of bilingual lexicons in three tasks: named entity recognition, part-of-speech tagging, and dependency parsing. the authors used bilingual lexicons to synthesize data and were able to use these synthesized data to achieve significant improvements in performance in all three of these tasks across 19 under-represented languages. these synthesized data were made by using bilingual lexicons to create synthetic sentences in the target language through word-to-word translation. word order and morphological differences between the source and target languages were not considered in this process, so the synthetic data may be quite different from true data in that language @xcite . bilingual lexicons have also been used to improve machine translation. despite the fact that approximately 70% of the world's languages have documentation in the form of bilingual lexicons @xcite , techniques involving bilingual lexicons in machine translation typically involve artificially generated lexicons. @xcite used terms extracted from two english-german lexicons and inserted these terms after their corresponding source words to improve translation quality. prior to this work, most work on neural machine translation used constrained decoding to integrate terminology, but @xcite used a black-box generic neural machine translation architecture that is directly trained to use an external lexicon provided during run-time, achieving increases of 0.2 to 0.9 in bleu scores. a more recent multilingual pre-trained method called mrasp @xcite also makes use of lexicons to improve translation, as demonstrated by experiments on 42 translation directions, including both low-resource and high-resource languages. in this method, a lexicon composed of unsupervised word alignments generated by muse @xcite is used to substitute words in the source sentence during pre-training. this is done to bring semantically similar words from different languages closer in the representation space @xcite . through a set of analytical experiments, the authors also showed that alignment information does help bridge any gaps between languages and improve translation quality. afromt @xcite is a machine translation benchmark for eight african languages, and the authors of this benchmark demonstrated that by extracting a dictionary from parallel corpora using eflomal foot_0 @xcite and taking word alignments that appear over 20 times to produce a bilingual lexicon, this lexicon can be used to produce code-switched monolingual data to aid in training machine translation models. the authors also experimented with iteratively creating pseudo-monolingual data in low-resource languages. most recently, @xcite showed that using bilingual lexicons to substitute source sentence words for their translations to create code-switched data and prepending translations of source words to source sentences are two techniques that can improve machine translation quality for low-resource and unsupervised languages. the authors used bilingual lexicons from panlex @xcite to do this. our work shows that there may be an even simpler approach to integrating lexical information when training neural machine translation systems. this makes use of existing bilingual lexicons without the need for inferring a bilingual lexicon from parallel data. ## data to explore whether lexicons can be simply used as additional parallel data in a low-resource setting in the context of asian languages, we believe it is important to experiment with a diverse set of languages. to that end, for this paper we collect data from 16 different formosan languages, a geographic grouping of austronesian languages indigenous to taiwan, which have a great deal of diversity in linguistic structure @xcite despite all being from a relatively small geographic area. to obtain this data, we downloaded lexicon entries and parallel sentences from an online dictionary (原 住 民 族 語 言 線 上 辭 典) foot_1 published by the indigenous languages research and development foundation (原住民族語言研究發展基金 會), an organization based in taiwan dedicated to research on taiwan's indigenous languages. we specifically consider the following languages: amis, atayal, bunun, kanakanavu, kavalan, paiwan, puyuma, rukai, saaroa, saisiyat, sakizaya, seediq, thao, truku, tsou, and yami. the dictionaries consist of words, phrases, and example sentences relating to daily life from each of these languages with their mandarin equivalents. we manually verified the data don't contain personal information or offensive content, and to the best of our knowledge, our use of these data is compatible with the original access conditions. while these data are available in an online dictionary format that can be searched, they were not immediately ready for computational use. one can download parts of the dictionary as pdf or odt files, but neither format is easy to use due to how the lexicon entries are organized. therefore, we downloaded pdfs from the website of the online dictionary, converted them into html using pdfminer foot_2 , and extracted data using beautiful soup foot_3 . each formosan word/phrase and its mandarin dictionary entry were extracted along with parallel sentence data included in this dictionary. these data are summarized in our studied formosan languages are all written using the latin script, unlike mandarin chinese. though there are a small number of formosan languages, their linguistic features are diverse @xcite . for example, word order, focus systems, auxiliaries, numerals, personal pronouns, compounding, affixation, and phonology all demonstrate great diversity across the formosan languages @xcite . most formosan languages are verb-initial, but some also have an svo (subject-verb-object) order and use different word orders depending on the situation. thao and atayal demonstrate little compounding, while tsou and bunun demonstrate rich compounding in their vocabularies. additionally, while some formosan languages distinguish between human and nonhuman numerals, some also do not. some formosan languages require auxiliaries, while others do not. we hope these examples, though limited, help illustrate how linguistically diverse the formosan languages are. one key point is that all of our studied languages are moribund or critically endangered. for example, thao was documented to have only four native speakers according to ethnologue in 2022. yet some other languages, such as amis, had roughly 30,000 speakers as of 2015 @xcite . this is still, however, a much smaller number of speakers than the most commonly spoken languages in the world, such as english and spanish, and to that extent, data for formosan languages are extremely lacking. it is also important to note that existing literature on these languages written in english may conflate the truku language and seediq language. this is due to the fact that the truku people (also known as the taroko people) were previously grouped with the seediq people by the taiwanese government. truku, in fact, has been described as a dialect of seediq ( lee et al., 2011 additionally, to test how our ideas work with an externally created lexicon, we used a seediq-mandarin lexicon of 1,556 entries from a seediq-mandarin dictionary foot_4 by temi nawi (曾 瑞 琳). this lexicon was kindly provided to us by darryl sterk. finally, to explore how our ideas work with non-formosan languages, we obtained parallel data and a lexicon for spanish and nahuatl, a language native to mexico. parallel data was obtained from americasnlp foot_5 and originally comes from axotol @xcite , a spanish-nahuatl parallel corpus. this dataset consists of 16,145 sentences in the training set, 672 sentences in the development set, and 1,003 sentences in the test set. a nahuatl-spanish lexicon containing 10,888 lexicon entries was obtained from aulex foot_6 . this lexicon consists of entries from three dictionaries: diccionario español-nahuatl by manuel rodríguez villegas, diccionario náhu- ## experimental setup as described in section 2, bilingual lexicons have played a significant role in some of the latest developments in machine translation. in contrast to existing work, we propose to simply treat bilingual lexicons as additional parallel data to explicitly inform the model of translations of different words. preprocessing data were tokenized using the unigram @xcite implementation of senten-cepiece @xcite . a vocabulary size of 8,000 and a character coverage rate of 0.99 were used to train our sentencepiece model. models our transformer models use six encoder and decoder layers with four attention heads each, a hidden dimension of 512, and a feedforward size of 1024, and a learning rate of 0.0005. our model was optimized using adam (kingma and ba, 2015) with hyperparameters β = (0.9, 0.98) and ϵ = 10 - foot_7 . a dropout rate of 0.3 and a weight decay of 0.0001 were used for regularization. each model for each language pair and direction was trained for two hours on three quadro rtx 8000 gpus. evaluation using the sacrebleu library 8 @xcite , we evaluated the translations outputted by our models with detokenized bleu @xcite on the test data from our parallel sentence data. we also used chrf @xcite to measure performance at the character level since most austronesian languages are agglutinative @xcite , meaning they make extensive use of affixes and are morphologically rich. for example, some languages such as thao @xcite and tsou @xcite have hundreds of prefixes. ## results and analysis we begin by creating a baseline by training models using only the parallel sentence data and one in which the model was trained using both the parallel sentence data and bilingual lexicon. to incorporate the bilingual lexicon during the training process, we added the lexicon entries in the source lan- having established the effectiveness of our technique, we present a sensitivity study regarding the amount of parallel and lexical data used to train our model, in an attempt to provide insights to 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 tables 3 and 4 summarize our sensitivity study results, showing how much lexical information is needed to attain performance improvements. we see that across all languages and directions in translation, there is a general trend of increases in translation quality as we add more lexicon entries. this is also indicated by the r 2 values shown in the tables, which indicate the correlation between the translation quality achieved and number of lexicon entries used during training. results for our sensitivity study regarding parallel data can be found in tables 5 and 6 for translation between mandarin and kanakanavu and ta-bles 7 and 8 for translation between mandarin and tsou. overall, we observe that more gains in translation quality were achieved by adding parallel sentences compared to adding lexicon entries. for example, for translation from kanakanavu to mandarin, adding 4,000 lexicon entries to the training data increased translation quality by 3.11 bleu points, whereas adding 4,000 parallel sentences to the training data increased translation quality by 6.22 bleu points. ## conclusion about 70% of the approximately 7,000 languages of the world have data in the form of lexicons @xcite , and these data are an incredibly valuable resource that can be used to improve low-resource machine translation. when parallel sentence data is limited for languages, it may be difficult for models to learn the translations of different words, and bilingual lexicons can be used to more explicitly teach which words correspond to one another. by adding lexical data during the training process, we achieved an average gain of 3.37 in bleu scores and 6.06 in chrf scores for translation between mandarin and 16 formosan languages, whose pairings can all be considered low-resource. these formosan languages demonstrate consid-erable variety in linguistic features and are considered the most diverse set of austronesian languages @xcite . by using existing lexicons and adding them to parallel data, we demonstrate how existing bilingual lexicons, a relatively inexpensive source of data, can be added as parallel data to improve translation quality for low-resource language pairs. we also show that adding more lexical data to parallel data during training tends to lead to higher translation quality. though adding lexical data is not as effective as adding more parallel data, lexical data are a much cheaper source of data that still generate significant improvements to model output. when using these lexicons to create pseudoparallel data, and using both the lexicon entries and this pseudo-parallel data during the training process, we achieved an average increase of 5.55 in bleu scores and 10.33 in chrf scores. adding lexical data was more effective than adding pseudo-parallel data, but adding both lexical data and pseudo-parallel data produced the most gains in translation quality. additionally, we demonstrate that these techniques also improve translation quality for one non-asian low-resource language pair, spanish-nahuatl. motivated by these results, we intend to continue exploring how these techniques work for other low-resource language pairs in the world. in the future, we'd also like to have humans evaluate our model outputs, though this remains a challenge due to the lowresource nature of the languages involved.