Neural Machine Translation for a Low Resource Language Pair: E nglish- B odo (2023)

Abstract: This paper represent a work done on Neural Machine Translation for English and Bodo language pair. English is a language spoken around the world whereas, Bodo is a language mostly spoken in North Eastern area of India. This work of machine translation is done on a relatively small size of parallel data as there is less parallel corpus available for english bodo pair. Corpus is generally taken from available source National Platform of Language Technology(NPLT), Data Management Unit(DMU), Mission Bhashini, Ministry of Electronics and Information Technology and also generated internally in-house. Tokenization of raw text is done using IndicNLP library and mosesdecoder for Bodo and English respectively. Subword tokenization is performed by using BPE(Byte Pair Encoder) , Sentencepiece and Wordpiece subword. Experiments have been done on two different vocab size of 8000 and 16000 on a total of around 92410 parallel sentences. Two standard transformer encoder and decoder models with varying number of layers and hidden size are build for training the data using OpenNMT-py framework. The result are evaluated based on the BLEU score on an additional testset for evaluating the performance. The highest BLEU score of 11.01 and 14.62 are achieved on the testset for English to Bodo and Bodo to English translation respectively.

## introduction english is a widely spoken language around the glove and generally forms an official language for most of the people. bodo is a local language of bodo people spoken in northeast of india (mostly in the state of assam). bodo language used the devanagari script for writing. recently, most of the work are going on for bodo language @xcite . since bodo is a low resource language, there are less digital resources for bodo language in the form of text corpus. this become quite challenging for natural language process-ing(nlp) task for bodo language like machine translation. here, we performed a machine translation for english to bodo and vice versa with limited amount of parallel corpus. this corpus is composed of parallel data from national platform of language technology(nplt), data management unit(dmu), mission bhashini, ministry of electronics and information technology and our own in-house created parallel corpus. preprocessing work that includes normalization,tokenization, subword tokenization are performed sequentially on both the languages and then trained on two transformer encoder-decoder models. these two encoder-decoder models varies among each other basically in terms of number of layers and hidden size. opennmt-py framework(a framework containing libraries and tool for performing machine translation) is used for build and training the model @xcite . normalization, i.e, conversion of all upper-case alphabets to lower-case is done on the english text. for tokenization of words in the text corpus of english and bodo language mosesdecoder and in-dicnlp library is been used respectively. subword tokenization which is widely used for processing of the tokenized text into sub-word level is performed. for subword tokenization three techniques have been used i.e, bpe(byte pair encoder) , sentencepiece and wordpiece subword. two vocabulary size of 8000 and 16000 subwords are created from the data generated from each of the subword tokenizations. the two transformer models are trained individually using data generated by each of the subword tokenization( bpe, sentencepiece and wordpiece subword) technique. training is done on both the directions(i.e, english to bodo and bodo to english) and using both the vocabulary sizes. the result are evaluated based on bleu using sacredbleu library. @xcite have proposed the transformer model, an encoder decoder model which is solely based on attention mechanism. in their expirement this architecture is found to be efficient than recurrent model for nlp task. this transformer architecture is used by us for our translation model. verma and @xcite in their paper have given a detail survey of nmt and the model and architecture used for nmt like lstm, encoder-decoder model, attention mechanism. they have addressed two problems in nmt task. one is poor performanes for long sentences which can be solved by attention mechanism and oov(out of vocabulary) which can be solved by subword bpe tokenization. and @xcite have given a detail specification of nmt models, architecture, tools, frameworks available for performing nmt tasks and other preprocessing libraries. @xcite in their paper have described an open-source toolkit for neural machine translation (nmt), opennmt. it is a framework for building neural machine translation model and it prioritizes efficiency and modularity. in our experiment we have used opennmt for building our nmt model. @xcite have performed an experiment to translate english to tamil language using neural machine translation model with word embedding and byte pair encoding tokenization. their model had been able to achieve a highest blue score of 8.33. indian languages like bodo, assamese and most of the northeastern indian languages are comparatively new to nlp research. in recent years various works including standardization, developement of tools and technology, corpus development, wordnets, annotations etc. have been carried out for bodo and assamese languages( sarma et al. (2010 2023 ). @xcite have build a machine translation system for bodo to english through the process of bodo to english machine transliteration system. they have used the phase based statistical machine translation method for developing their model. @xcite and @xcite in their respective papers have shown some techniques for preprocessing and modeling of resources that are required for performing a neural machine translation training for english-bodo and english-assamese language pairs respectively.as the above research dicussed briefly about preprocessing of the language pairs and later in @xcite and @xcite highlights multiple nmt models with varing hyperparameters for english-bodo and english-assamese respectively. ## conclusion in this paper we have performed experiments on neural machine translation for english to bodo and bodo to english using transformer encoder decoder model. here we have trained the model using 92410 parallel sentences which we have have collected from national platform of language technology(nplt), data management unit(dmu), mission bhashini, ministry of electronics and information technology and our own in-house created parallel corpus we have performed preprocessing such normalization, tokenization and subword tokenization. three methods of subword tokenization have been explored(i.e, bpe, sentencepiece and wordpiece subword) trained individually with two different vocabulary sizes of 8000 and 16000 on both model 1 and model 2. for english to bodo translation model 2 trained with a vocabulary size of 8000 from bpe subword tokenized text data gives the highest bleu score of 11.01 and for bodo to english translation same model 2 trained with a vocabulary size of 8000 but from wordpiece subword tokenized text data gives the highest bleu score of 14.62. since from our expirements we have seen that bpe subword tokenization has given the best result for english-to-bodo translation and wordpiece subword tokenization has given the best result bodo-to-english translation, this can be a potiential objective of further reasearch to identify the affect of different subword tokenization methods for different language pairs.we observed that model 2 which has more number of layers, hidden sizes etc. than model 1 gives a higher result among both the model. in most of our cases, training the models with 8000 vocabulary sizes gives higher bleu score than training the same model with 16000 vocabulary sizes. also in all the cases, the same model trained with same subword data with same vocabulary sizes in bodo to english direction gives a higher bleu score than english to bodo direction.