Machine Translation for a Very Low-Resource Language - Layer Freezing Approach on Transfer Learning (2022)

Abstract: This paper presents the implementation of Machine Translation (MT) between Lambani, a low-resource Indian tribal language, and English, a high-resource universal language. Lambani is spoken by nomadic tribes of the Indian state of Karnataka and there are similarities between Lambani and various other Indian languages. To implement the English-Lambani MT system, we followed the transfer learning approach with English-Kannada as the parent MT model. The implementation and performance of the English-Lambani MT system are discussed in this paper. Since Lambani has been influenced by various other languages, we explored the possibility of getting better MT performance by using parent models associated with related Indian languages. Specifically, we experimented with English-Gujarati and English-Marathi as additional parent models. We compare the performance of three different English-Lambani MT systems derived from three parent language models, and the observations are presented in the paper. Additionally, we will also explore the effect of freezing the encoder layer and decoder layer and the change in performance from both of them.

## introduction machine translation started way back in the 1950s as a way to bridge the communication gap. the techniques are broadly classified in three types (a) rule-based machine translation(rbmt) @xcite statistical based machine translation(smt) @xcite and (c) neuralbased approaches (nmt). warren weaver cre-figure 1 : distribution of lambani language in the state of karnataka ated the first computer-generated machine translation @xcite during the 1980s by using statistical methods using 'shannon's information theory' (stone). in the last couple of years, neural-based machine translation has achieved state-of-the-art performance where large amounts of parallel data are available. with the introduction of the encoderdecoder-based architecture @xcite , there was a surge of interest and a lot of research has been conducted. however, it was quickly realized that these initial systems require a huge amount of data to get a performance close to that of a smt system. @xcite . transfer learning has proved successful for low resource settings @xcite @xcite and achieves higher translation performance. in this paper, we will specifically be focusing on nmt although transfer learning has been used for smt in the past. specifically for low-resourced languages smt seem to give better performance in case of domain mismatch @xcite in this paper, we focus on lambani language @xcite which is generally spoken by the banjaras @xcite and study how the language draws its influence from various other languages. we show how morphological similarity can improve the performance of a language. we focus on three different languages and how are they related to lambani. however, it is a major challenge to collect a large amount of data for languages which are not spoken by a lot of people. despite the recent emphasis on low resource languages, we are not aware of any research that has done any work in the lambani language. the paper is organized as follows. a summary of the background work is given in section 2. the proposed approach is explained in section 3. the details of the dataset used are given in section 4. the effect of layer freezing is presented in section 5. ## background in this section we give an overview of the transfer learning approach in the context of machine translation. ## proposed approach in this proposed work we will first train a parent pair containing a large number of sentences for a given number of iterations and then switch to the child language pair without changing any of the hyperparameters. then subsequently the performance is improved by freezing some of the layers where weights of some layers are frozen. transfer learning in machine translations was first proposed by @xcite . we will be applying the same principle in this work . the lambani language has no script of its own and it is generally written in kannada script. whereas, marathi and gujarati generally follow devanagari script. to avoid script mismatch we will be transliterating both marathi and gujarati to kannada script. to the best of our knowledge this work demonstrates the effectiveness of transfer learning for very low resource indian tribal language. the novel part of this paper is that we will not be sharing any vocabulary instead we will use distinct vocabularies for the parent and child models. a shared vocabulary will not work in our case as some of the parent models don't share lexical features with the child model. we will also incorporate encoder and decoder layer freezing and how they impact the performance of our child model. during our training, we train our nmt model on high resource data and this is called our parent model. then we will be using the parent model to train the child model on lowresource data using the transfer learning approach. ## dataset in our experiment, we are be mainly working on a low-resource dataset. we consider lambani as a very low-resource language reason being that no text-based resources are available to the best of our knowledge. while kannada, marathi, and gujrati are considered to be medium resources languages. the size of the dataset is given in ## future work although there may have been a couple of research on transfer language of related and unrelated languages there is very little research figure 5: bleu score curves on the val set for en-kn as parent figure 6: bleu score curves on the val set for en-mr as parent as to why transfer learning is giving better results for related languages from a linguistic perspective. as in our case lambanis a nomadic tribe before they settled in the modern state of karnataka. as a result, the language is morphologically rich and may share some linguistic similarities with other language. according to @xcite adding noise to the training data has improved neural machine translation. the same idea can be applied to our model. we can randomly drop words from the training data and replace them with filler words in order for the model to learn better. noisy sentence help in learning as it makes it harder to predict translation. ## conclusion our experiment is limited to a transfer learning method between closely related languages. from our experiments, we are seeing much better performance when similar languages are taken for transfer learning while for unrelated languages we are not seeing a drastic change in bleu @xcite score which may be because of our dataset size of all the parent models is almost similar. we have further improved our model performance by incorporating encoder freezing and reached a performance improvement of +3 over the en-mr-la model. from our experiments we also notice that freezing the decoder is reducing the performance. this may be because the decoder needs more data than an encoder.