Transfer learning in low-resourced MT : An empirical study (2023)

Abstract: Translation systems rely on a large and goodquality parallel corpus for producing reliable translations. However, obtaining such a corpus for low-resourced languages is a challenge. New research has shown that transfer learning can mitigate this issue by augmenting lowresourced MT systems with high-resourced ones. In this work, we explore two types of transfer learning techniques, namely, crosslingual transfer learning and multilingual training, both with information augmentation, to examine the degree of performance improvement following the augmentation. Furthermore, we use languages of the same family (Romanic, in our case), to investigate the role of the shared linguistic property, in producing dependable translations.

## introduction for any machine translation (mt) system to produce reliable translations, it needs to be trained using a large and good-quality parallel corpus. although, these corpora are abundant for european languages with high digital presence (where both the participating languages are high-resourced (hr)), finding one for low-resourced (lr) languages is difficult (one of the participating languages is lr). this leads to the development of a small-sized parallel corpus, which when used in training does not produce robust mt systems and translations. over the years, many research works have focused on the effect of transfer learning on mt. this translates from experiments where lr language data have been augmented using hr data to produce better outputs when compared to vanilla mt models trained using lr languages only @xcite . furthermore, training hr and lr mt models together have also given considerable improvements in the quality of mt output. this paradigm for multilingual translation can be compared naively to how human individuals learn new languages and acquire new languages. in the indian context, for instance, a natural english speaker who is conversant in hindi could pick up marathi more quickly than someone completely ignorant of any indic languages. second language acquisition is associated with cognitive rewiring and anatomical alterations in the human brain, as proposed by @xcite . furthermore, schepens's empirical research @xcite shows that bilingual speakers' ease of acquiring a third language is closely correlated with the languages' distance from one another. the authors noted that several factors, including anthropological development, speaker geography, vocabulary exchange, syntactic structural similarities, etc., influence how simple it is to acquire a new language. @xcite demonstrated in a different work the connection between word usage frequency and the development of indo-european languages. the previous works inspired us to look into the cases where lr mt models may gain information when trained incrementally on already trained hr mt models (where the participating languages belong to the same language family), with information augmentation, using various transfer learning approaches. we used two transfer learning approaches to test this. the first one, called the cross-lingual transfer technique with information augmentation, initially trains an hr mt system. after the training is over, the weights of the hr model are saved. thereafter, an lr mt system is trained, where the saved weights of the hr model are then used to initialize the model training. furthermore, before sending the information to the decoder, the input sentence is passed through both encoders (both hr and lr) and the resulting context vectors are concatenated. this method allows the transfer learning model to gain from already trained hr model weights as well as to benefit from the already trained hr encoder vector. in the second approach, called multilingual learning with information augmentation, a multilingual hr mt system is trained with multiple hr language pairs. similar to the previous approach, the weights are saved again. the same process, including the hr weights and concatenation of the hr encoder and lr encoder vector, is followed before sending information to the decoder. the lr model, in this approach, benefits from the shared linguistic property of the participating hr languages that belong to the same language family. ## related work the training of nmt systems without the use of parallel or comparable data has been extensively studied. such studies typically combine designs incorporating several encoders and decoders @xcite , heavily rely on cross-lingual embedding, and perform iterative back translations @xcite . additionally, it has been discovered that training nmt systems in multiple languages enhances translation performance @xcite @xcite . a single encoder that is shared by all languages and a decoder for each language were utilised by @xcite and @xcite . a single shared attention mechanism is used by @xcite to propose multi-way cum multilingual nmt with multiple encoders and decoders. @xcite developed a more straightforward yet efficient method that only required one encoder and one decoder. all of the parallel data were combined into a single corpus after certain unique tokens were added at the start of each sentence. a transfer-learning strategy was put up by @xcite to share lexical and sentence-level representations from several source languages into a single target language. a similar architecture was described by @xcite who trained a high-resource language pair (the parent model) first. the weights from the parent model were initially initialised in the child model, which was trained on the language pair with limited resources. they assisted with the english translation of hausa, turkish, uzbek, and urdu by using a french-english parent model. to the best of our knowledge, there hasn't been any published research on how, when all languages belong to the same language family, a system trained in an hr language can improve the translation output of a system trained in an lr language. furthermore, we didn't merely use the hr model's weights to initialise the lr model. in addition, the encoder vectors of the lr model were concatenated with the hr model's vectors for each source sentence, fed with the same sentence, and then sent to the decoder for prediction. ## data as mentioned earlier, we wanted to test whether mt models trained using hr languages, benefit an lr mt model when used in accordance to transfer learning approach. furthermore, we wanted to keep the participating languages belonging to the same language family, so that we can investigate the concept of shared linguistic property. for this, we stuck to experimenting with the romanic language family, which consists of languages like french, italian, spanish, portuguese and romanian. for experimentation purposes, we considered (french, italian, spanish, and portuguese) -english language pairs as hr and romanian -english language pairs as lr for the second transfer learning approach. on the contrary, for the first approach of transfer learning, we considered french -english as the hr language and romanian -english as the lr language. for this, the amount of data for hr language pairs was adjusted to 100k and 50k for the lr language pair. we named these experiments as mt samefamily as a whole. moreover, to find whether languages from the same language family only aid in better translations, we experimented with a separate language family (slavik, in our case), which consisted of bulgarian, czech, polish and slovak languages. these languages again, were considered as hr languages and 100k parallel sentences (hr -english) of these were considered for the second transfer learning approach. for the first transfer learning approach, we considered bulgarian -english language pair as the hr language. we named these experiments as mt difffamily as a whole. the parallel corpus for the above-mentioned languages was extracted from the europarl project foot_0 . ## methodology embedding layer lstm 1 lstm 2 embedding layer lstm 1 lstm 2 dense layer words of source language words of target language (at time t) words of target language (at time t+1) c encoder decoder typical encoder-decoder architecture encoder decoder portuguese-english hr mt model encoder decoder romanian words c c c hr c lr query lr mt model weights of lr mt initialized with hr mt weights english words schur product italian-english spanish-english french-english augmentation. for this approach, the hr mt model (essentially a seq-to-seq model based on lstm cells) was first trained using the french -english language pair, for the mt samefamily experiments and bulgarian -english language pair, for the mt difffamily experiments. the hyperparameters for training these models were as follows. the batch size was set to 64, the number of epochs was set to 100, the activation function was softmax, the optimizer chosen was rmsprop and the loss function used was sparse categorical cross-entropy. the learning rate was set to 0.001. after the hr mt models were trained, the weights were saved. thereafter, the lr mt model was trained using the romanian -english parallel corpus. the weights of the lr mt model were initialized with the saved weights of the hr mt model. this time around, the source romanian sentence was fed to the encoder of the lr mt model. at the same time, this sentence was also sent to the trained encoder of the hr mt model. both resultant context vectors, c hr and c lr were concatenated using schur product operation. this concatenated context vector was then sent to the decoder part to complete the training process of the lr mt model. the hyper-parameters for this model were kept alike with the hr mt model. for training the multilingual learning technique with information augmentation, the hr mt model was trained using a mixture of language pairs, with sentences belonging to the same language family for mt samefamily and different language family for mt difffamily . for this, the sentences from language families were coupled together. this gave rise to two source corpora, that consisted of a mixture of french, spanish, italian and portuguese languages and bulgarian, czech, polish and slovak languages. the target corpus consisted of all english sentences. the same procedure of training hr mt models first, saving the weights, passing romanian sentences through the encoder of the lr mt model and hr mt model to get concatenated context vector, etc., was followed. the same hyper-parameters were used to train the second transfer learning approach as well. the whole architecture followed for training both our approaches has been depicted in figure 1 . also, a vanilla mt model, using seq-to-seq architecture was trained using 50k sentences of romanian -english and this acted as our baseline system through which we could investigate the performance gain via transfer learning. the same hyper-parameters, as mentioned above, were used to train this baseline model. ## results and discussion the results of the above experiments were quantified using automated mt evaluation metrics. for the automated metrics, we used bleu @xcite , chrf @xcite and ter @xcite to evaluate the translation quality. the results have been shown in for experiments of mt difffamily , we see that both the transfer learning approaches do not produce improvements in the quality of translation output. ## conclusion and future work in this work, we see the benefit of using transfer learning approaches for low-resourced machine translation. in our case, we experimented with the romanic language family, which consisted of languages like french, italian, spanish, portuguese and romanian. another set of experiments with the salvik language family, which consisted of languages like bulgarian, czech, polish and slovak were also performed. while it is a known fact that transferring knowledge of a high-resourced mt model into a low-resourced mt model aids in producing better translation quality, we wanted to test the same concerning a language family. for our experiments, we considered languages of the same family (french -english, spanish -english, italian -english and portuguese -english) and different language family (bulgarian -english, czech -english, polish -english and slovak -english) as high-resourced and romanian -english as low-resourced. experimentation, using two types of transfer learning approaches shows that performance (bleu, ter and chrf) does increase when languages of the same language family are used in the transfer learning setting, as this approach takes advantage of the common linguistic property shared by the languages of a same family.