Geographical Distance Is The New Hyperparameter: A Case Study Of Finding The Optimal Pre-trained Language For E nglish-isi Z ulu Machine Translation. (2022)

Abstract: Stemming from the limited availability of datasets and textual resources for low-resource languages such as isiZulu, there is a significant need to be able to harness knowledge from pre-trained models to improve low resource machine translation. Moreover, a lack of techniques to handle the complexities of morphologically rich languages has compounded the unequal development of translation models, with many widely spoken African languages being left behind. This study explores the potential benefits of transfer learning in an English-isiZulu translation framework. The results indicate the value of transfer learning from closely related languages to enhance the performance of low-resource translation models, thus providing a key strategy for low-resource translation going forward. We gathered results from 8 different language corpora, including one multi-lingual corpus, and saw that isiXhosa-isiZulu outperformed all languages, with a BLEU score of 8.56 on the test set which was better from the multi-lingual corpora pre-trained model by 2.73. We also derived a new coefficient, Nasir’s Geographical Distance Coefficient (NGDC) which provides an easy selection of languages for the pre-trained models. NGDC also indicated that isiXhosa should be selected as the language for the pre-trained model.

## introduction neural machine translation aims to automate the translation of text or speech from one language to another utilising neural networks @xcite . consequently, the performance of neural machine translation (nmt) models is highly dependent on the availability of large parallel corpora to provide sufficient training data. low-resource languages which are under-represented in internet sources lack suitable training corpora and therefore suffer from limited development, obtaining poor translation performance. this phenomenon is exacerbated by a lack of content creators, dataset curators and language specialists, resulting in barriers at many stages in the translation process @xcite @xcite . therefore, due to the historical focus on dominant languages such as english in the development of neural machine translation (nmt) models, lowresource and morphologically complex languages remain a challenge for current translation systems @xcite . due to limited resources in terms of both computational expense and available datasets, it is vital to be able to leverage knowledge from current pretrained models to provide more effective solutions. therefore, in this investigation, the effects of transfer learning from closely related languages, as well as comparison with high-resourced languages for pre-trained scenario, is explored in the context of english to zulu translation. furthermore, this study derives the nasir's geographical distance coefficient. geographical distance (gd) @xcite has been studied for various scientific research areas @xcite @xcite as it provides deep insights in many aspects. we will also use gd as a hyperparameter for an attempt to get a language for a pre-trained model in an effective and with a o(n) complexity. although there are many ways to find gd, we will use literal approximation of distance in kilometers and suggest the techniques in future directions. ## methodology this investigation evaluates several models pretrained on different language pairs, both low-and high-resourced, on a recently release english-zulu parallel corpus. the dataset utilized to fine-tune and benchmark the models is discussed below. ## results each model was benchmarked on the test set using the bleu @xcite score as tabulated in in fig. 1 below, we can observe that the mar-ianmt model pre-trained on the english-xhosa dataset outperforms all other models by a good margin, obtaining a final bleu score of 8.56. this result suggests that the morphological similarities between the isizulu and isixhosa languages plays a strong role in the benefits attained through finetuning. following identification of the optimal model, the marianmt model pre-trained on the en-xh dataset was further fine-tuned for 75 epochs on umsuka dataset, giving a final optimal bleu score of 17.61 on training set and 13.73 on test. ## analysis we now present an analysis of the results in light of both the underlying theory and previous literature. in order to further understand the effects of pre-training on different languages, the datasets used for pre-training of the marianmt models were inspected. notably, although the number of sentences in english-xhosa dataset is in order of magnitudes less than romance languages corpus but still performs better. this justifies our hypothesis and opens up a path to effective fine-tuning through the knowladge of morphologies and not by adding multiple languages into a single corpus. arabic and french having approximately 5 and 23 times more data also suggests the above mentioned hypothesis that with closer gd and lesser data is much better, in many ways, than larger data and farther gd. other bantu languages that were selected, kiswahili and chishona performed almost similar to arabic and french with order of magnitudes of lesser data which suggests that even if they are language(s) bleu( val not as similar to isizulu, the distance being very close to where isizulu is spoken tends to have a great impact. we speak similar languages in neighbouring cities and countries which should have an effect on the model and so the result suggests. twi and luganda, having very less data and higher gd, gives us very poor results. from ## impact statement the potential impacts of this investigation can be explored in light of the possible contributions, risks and societal impact. ## conclusion and future directions english-isizulu translation has historically obtained poor results on translation benchmarks due to a lack of high-quality training data and appropriate tokenization schemes able to handle the agglutinative structure of isizulu sentences. in this investigation, the challenges of isizulu translation in terms of both morphological complexity and a lack of textual resources are explored using the recently released umsuka english-isizulu parallel corpus. in order to investigate the effects of the impact of the pre-trained model selected for transfer learning, several models were fine-tuned and benchmarked on the umsuka dataset. mariantmt models pre-trained on english-xhosa, english-swahili, english-shona, english-twi, english-luganda, english-arabic, english-french and english-multilingual romance languages, respectively. the study found that the pretrained english-xhosa model attained the optimal results with a handsome margin. thus, the results indicate that transfer learning is particularly effective when languages are within the same sub-family while transfer learning is less effective when the model is pre-trained on a more distantly related language, no matter the size of the data to an extreme extent. we have also introduced a novel nasir's geographical distance coefficient which will help researchers find a language for pre-trained model effectively and will result in using less resources. therefore, this study motivates careful choice of the pre-trained model used for transfer learning, utilising existing knowledge of language family trees, to promote improved performance of low resource translation. in addition, we have opensourced foot_2 our best model which was fine-tuned for 75 epochs using the original marianmt model pretrained on the english-xhosa language pair, obtaining a final bleu score of 17.61 on train while 13.73 on test set. we have also gathered all model cards for the models that were used for further experimentation. this study yeilds promising future directions as the experiment was done on only 8 corpora. we suggest to increase the number and observe the derivation of the result. we also suggest to combine bantu language as one multi-lingual corpora and observe the result. the experiment has been done on a novel umsuka parallel corpora, the study should extend to more common benchmarks. this study should extend to different low-resourced languages of different continents of our world. we have derived a formula that takes into the account just the distance and the size of corpora, a promising research would be to derive a formula that takes morphologies and/or phonologies and fives a distance based on that. with ngdc at hand, it motivates to create a framework where one enters a target language, a d max and a value for weight coefficient c and gets desirable models to train on. there are many precise ways of finding gd, such as lambert's formula @xcite and vincenty's formula @xcite which may enhance ngdc's performance. it also opens up ways to introduce morphology in the formula, which we expect it to improve the overall selection of the models.