A Comparison of Different NMT Approaches to Low-Resource D utch- A lbanian Machine Translation (2021)

Abstract: Low-resource languages can be understood as languages that are more scarce, less studied, less privileged, less commonly taught and for which there are less resources available (Singh, 2008; Cieri et al., 2016; Magueresse et al., 2020). Natural Language Processing (NLP) research and technology mainly focuses on those languages for which there are large data sets available. To illustrate differences in data availability: there are 6 million Wikipedia articles available for English, 2 million for Dutch, and merely 82 thousand for Albanian. The scarce data issue becomes increasingly apparent when large parallel data sets are required for applications such as Neural Machine Translation (NMT). In this work, we investigate to what extent translation between Albanian (SQ) and Dutch (NL) is possible comparing a one-to-one (SQ↔AL) model, a low-resource pivot-based approach (English (EN) as pivot) and a zero-shot translation (ZST) (Johnson et al., 2016; Mattoni et al., 2017) system. From our experiments, it results that the EN-pivot-model outperforms both the direct one-to-one and the ZST model. Since often, small amounts of parallel data are available for low-resource languages or settings, experiments were conducted using small sets of parallel NL↔SQ data. The ZST appeared to be the worst performing models. Even when the available parallel data (NL↔SQ) was added, i.e. in a few-shot setting (FST), it remained the worst performing system according to the automatic (BLEU and TER) and human evaluation.

## introduction there are more than 7000 languages worldwide, with over 40% of the languages being endangered with less than 1000 speakers. on the other hand, roughly 35% of the world population, close to 3 billion people, account for only 3 languages: english, mandarin chinese and hindi @xcite . this language division results in an inequality in literary resources available per language. languages with a significant amount of literature and speakers are known as high-resource languages in the field of natural language processing (nlp), whilst languages that lack these resources are known as low-resource languages. @xcite 2) define low-resource languages as "languages that have a low population density, are undertaught or have limited written resources or are endangered". these languages are, therefore, not properly represented through literary media, resulting in an insufficient amount of training data availability. one such low-resource language is albanian. albanian is spoken by approximately 8 million people in the world, the majority of which reside in albania and kosovo where the language is native to @xcite . the language, however, is not limited to albania and kosovo, but extends further into other parts of the balkans, such as montenegro, macedonia and italy as well as switzerland, places where albanian is recognized as a minority language @xcite @xcite . both @xcite and @xcite discuss the impact of the ottoman rule on albanian literature, a period during which publications in albanian were forbidden, resulting in little to no development of the written culture. as discussed by @xcite , this led to a limited number of resources in albanian, despite the vast number of speakers, books in albanian were not commonplace until the late 19th century. the impact of which, bred a limited amount of albanian sources adequately translated into other languages. a lack of readily available albanian training data makes developing statistical mt (smt) and nmt methods difficult, as these methods require significant amounts of parallel data between language pairs in order to create useful mt systems @xcite . additionally, parallel corpora are often times domain-specific, leading to poor performance when deploying mt models for translating material outside of the trained domain @xcite . access to knowledge is a key driver for developing countries to progress in terms of educational, scientific, and societal advancement @xcite . as such, creating opportunities to acquire general knowledge in languages native to developing countries could accelerate the development of their population. one of the most commonly known contributors to online open-access knowledge is wikipedia @xcite . however, in terms of accessibility there is a significant lack of articles in non-major languages. for example, there are over 6 million english articles and more than 2 million articles in dutch, while articles written in albanian only account for approximately 82 thousand articles. 1 while the access to knowledge can depend on multiple factors such as, the ability to read and understand english, the access to a stable internet connection, the lack of albanian training data for nmt models suggest that online literary resources in the language are scarce. 2 being able to automatically translate wikipedia articles to a low-resource language offers open-access knowledge to a wider array of people, while allowing these users to improve on the automatically generated translations. consequently, these improvements can be propagated back to the nmt model, which can help translate future articles more accurately. in this work, we compare a one-to-one nmt model and two low-resource nmt approaches to translation from dutch (nl) to albanian (sq), a low-resource language pair. by automatically and manually evaluating the translations, we aim to provide insights into how accurately nl↔sq models can translate. we furthermore explore how the addition of direct parallel nl↔sq data affects the performance of the zst model, since often small amounts of parallel data are available. the main research questions can be formulated as follows: (a) "to what extent are low-resource direct one-to-one nl↔sq, pivot-based and zero/few-shot nmt models able to accurately translate and how do they compare?" and (b) "how does adding parallel nl↔sq data affect the performance of the zst model?". the performance of the models is evaluated and compared using automatic metrics (bleu and ter) as well by providing a more detailed human evaluation of 100 random sentences for all models evaluated by three native albanian speakers. ## discussion according to the automatic and human evaluation, the pivot approach performs best, however, as evidenced in a major point that requires addressing, is the difficulty of translating any language when the context is lacking. in this specific case the data used for the sentences came from movie translations, where context is inherently significant. two examples that highlight the difficulty will be explored and discussed below. ## conclusion in this paper, three approaches to @xmath0: a one-to-one direct model and two approaches specific to low-resource settings, pivot-nmt and zst, including fst -where small amounts of parallel data was added to the zst models. from our experiments it results that the pivot approach outperformed the others in terms of the automatic (bleu & ter) and human assessment. additional experiments were conducted where small amounts of parallel nl-sq data was added to the zst training data leading to improvements, approaching the results obtained using english as a pivot. additionally, zst/fst has some advantages over pivot-based mt in terms of efficiency as it only requires the training of one model. in future work, we would like to further explore how parallel data affects the performance of zst models and experiment with different, morphologically richer pivot languages since english does not capture many of the specific linguistic properties of albanian (gender, cases...).