Can We Use Word Embeddings for Enhancing G uarani- S panish Machine Translation? (2022)

Abstract: Machine translation for low-resource languages, such as Guarani, is a challenging task due to the lack of data. One way of tackling it is using pretrained word embeddings for model initialization. In this work we try to check if currently available data is enough to train rich embeddings for enhancing MT for Guarani and Spanish, by building a set of word embedding collections and training MT systems using them. We found that the trained vectors are strong enough to slightly improve the performance of some of the translation models and also to speed up the training convergence.

## introduction in recent years the performance of machine translation systems has grown alongside with the rise of neural architectures @xcite that infer the translation patterns while consuming a huge amount of data at training time. however, this high performance is hard to achieve when one (or both) of the languages is considered a low-resource language @xcite . that is the case for guarani, an indigenous language spoken by nearly 10 million people in south america. it has the characteristic of being one of the few indigenous languages used for daily communication, both by people who identify with indigenous ethnicity as well as people who do not. according to the paraguayan census office almost 70% of paraguayans speak some form of guarani at home foot_0 , but despite this, it remains a low-resource language in the nlp community @xcite , and the existing attempts at building machine translation systems for this language have not achieved very high results yet. @xcite found that using pretrained word embeddings could be useful when building ma-chine translation systems for low-resource scenarios. considering the scarcity of guarani-spanish parallel text, the aim of this work is to evaluate if it is possible to enhance a mt system by incorporating word embeddings built with the available monolingual data. in order to do this, we first trained a set of word embedding collections and selected the best of these models according to some intrinsic tests. finally we trained machine translation experiments using the different embeddings and compared them to the base scenario where no pretrained embeddings were used. the intrinsic tests and other resources used in this paper are available on github foot_1 . ## related work although there have been some efforts on developing resources for guarani, it remains largely under-explored in nlp. the current reference corpus for guarani is coreguapa (secretaría de políticas lingüísticas del paraguay, 2019), it can be queried online but not be downloaded. other resources include a spanish-guarani parallel corpus built from news sites and blogs @xcite , two corpora for sentiment analysis @xcite , and a small universal dependencies corpus of the mbya guarani dialect @xcite . except coreguapa, which cannot be downloaded, all of these resources are rather small for building accurate statistical models. interest towards machine translation for indigenous languages of the americas has increased lately. an important antecedent is the first workshop on nlp for indigenous languages of the americas (americasnlp) @xcite , which organized a shared task on mt from spanish to several indigenous languages, including guarani, with several participants. the test set for this shared task was a subset of the xnli corpus @xcite translated to all languages. however, guarani-spanish machine translation still remains under-explored. there are some works that take into account the lack of available data @xcite @xcite , or try to use the rich guarani morphology to enhance the translation results @xcite . the use of word embeddings to enhance machine translation in low-resource scenarios has been previously explored @xcite , obtaining good results overall. they report that using pre-trained embeddings for both the source and target languages seem to improve results for translating low-resourced languages, but the improvement is much lower for languages with large amounts of data. furthermore, @xcite explores alternatives to include pre-trained embeddings in mt systems for a morphologically rich language, and (nguyen and chiang, 2017) uses a transfer learning approach for enhancing translation for a low-resource pair, but considering data from other related low-resources pairs as well. ## word embeddings in a previous work @xcite we carried a first round of experiments with guarani word embeddings, collecting text from news sites, tweets and the guarani wikipedia foot_2 . we classified each tweet in one of three categories (a: very reliable, b: reliable, and c: unreliable) according to the probability of being in guarani using a heuristic based on the number of guarani tokens from a frequent words list. finally, for evaluating the then trained embeddings, we also presented two sets of intrinsic tests based on the original tests from @xcite . one of them is a translation of the original capital-common-countries (ccc) set, while the other is a new set for family relations, inspired in the original one. in the current work, we collected more data from the different sources and added datasets such as the bible foot_3 and the book of mormon foot_4 . we also translated the classic similarity test mc-30 @xcite to guarani in order to have another intrinsic test to perform (in addition to the family and capital-common-countries tests). we trained a set of 24 different word embedding models in guarani with different configurations. all of them were built using the gensim library @xcite implementation of the word2vec c-bow algorithm @xcite . the configurations differ in how much text was used (see below), the embeddings size (150 or 300) and the window size (6, 7 or 8). the number of tokens used in the different experiments varies between 1.9m and 2.7m depending on the different data sets we use, as shown in ## machine translation experiments we carried a series of machine translation experiments to compare the use of randomly initialized embeddings with the use of different pretrained embedding configurations. all experiments were done using opennmt foot_6 with its default configuration, an encoder-decoder model implemented with stacked lstms and an attention model, so that the difference between experiments would only be the embeddings initialization. for those models using pre-trained word embeddings we had to choose both the spanish embeddings and the guarani embeddings. for spanish we chose a collection of size 300 trained by azzinnari and martínez (2016) using a corpus of 6 billion words. due to limitations of opennmt, the guarani embeddings size must also be 300. therefore we chose some of the twenty-four models trained according to their size (300), their spearman's correlation score for the mc-30 test (see we then chose, for each model, the checkpoint that maximized the chrf metric for the dev set (dev2020+dev2021). the test results will be reported over the test set from @xcite (test2020), the test partition of our own parallel set (test2021), and the dev and test sets from @xcite (anlp dev and anlp test), using the bleu @xcite and chrf @xcite scores. ## conclusions the results obtained in our experiments show that -with the currently available data -we can start to see some improvements when using pre-trained embeddings; at least in the gn-es direction. the performance of the gn-es models that used pretrained embeddings was slightly better than the performance of the one that did not use them. additionally, the developed systems converge faster when using pretrained embeddings, which is especially useful in the scenario that is common for low-resource research labs, that of having little computing power. however, in the es-gn direction the results were more mixed, which is aligned with the conclusions of @xcite . there are still many lines to explore. first, trying other methods and algorithms for building embeddings such as fasttext, which could be better for morphologically rich languages such as guarani @xcite . second, we must explore the different opennmt configuration possibilities. we could also use backtranslation techniques as well, such as the approach explored by @xcite , the winning system in americasnlp shared task. finally more diverse text is needed, considering the difference observed while evaluating over the americasnlp sets. this diversity is also needed for improving the word embeddings performance. the great differences between both analogy tests suggests that the words in the capital-common-countries test might not be suitable for guarani, perhaps due to the topics covered in paraguayan news which refer mainly to countries in the region.