A Baseline System for K hasi and A ssamese Bidirectional NMT with Zero available Parallel Data: Dataset Creation and System Development (2023)

Abstract: In this work we have tried to build a baseline Neural Machine Translation system for Khasi and Assamese in both directions. Both the languages are considered as low-resourced Indic languages. As per the language family in concerned, Assamese is a language from IndoAryan family and Khasi belongs to the MonKhmer branch of the Austroasiatic language family. No prior work is done which investigate the performance of Neural Machine Translation for these two diverse low-resourced languages. It is also worth mentioning that no parallel corpus and test data is available for these two languages. The main contribution of this work is the creation of Khasi-Assamese parallel corpus and test set. Apart from this, we also created baseline systems in both directions for the said language pair. We got best bilingual evaluation understudy (BLEU) score of 2.78 for Khasi to Assamese translation direction and 5.51 for Assamese to Khasi translation direction. We then applied phrase table injection (phrase augmentation) technique and got new higher BLEU score of 5.01 and 7.28 for Khasi to Assamese and Assamese to Khasi translation direction respectively.

## introduction while neural machine translation (nmt) has achieved state-of-the-art in machine translation domain, the inherent nature of huge data requirement of neural systems makes it extremely difficult to get higher score in tasks where the data is scarce. to achieve high bleu @xcite score in the translation task, the languages under study must have a good amount of parallel data. for languages where such parallel data is very low or not available at all, it becomes challenging to create a decent nmt system for these languages. in this work, we have engaged our research in studying how one can proceed to develop an nmt system for languages where no parallel data is available and the languages are completely unrelated. we will briefly describe about the languages in section 2. in section 3 we explored existing works. section 4 describe the process of data collection and generation. section 5, 6 and 7 shows the experiments, results and finally conclusions respectively. ## about the languages assamese is an official language in india among 22 languages as declared in the eighth schedule of indian constitution foot_0 . assamese belongs to the indo aryan language family and is mainly used in the north eastern part of india with more than 16 million speakers @xcite and is considered as a lingua franca in that region. but the language has been recognised as a lowresourced language for machine translation task @xcite . assamese is written using the bengali-assamese script foot_1 or sometimes also called assamese-bengali script @xcite . another language, 'khasi', is mainly spoken in the north eastern state of india, meghalaya. khasi is not yet included as a scheduled language in india. it is a resource poor language. it belongs to the mon-khmer branch of austroasiatic language @xcite , which is a language family found mainly in southeast asia and the eastern part of the indian subcontinent. the other languages in the khasic group of the shillong plateau, such as pnar, lyngngam, and war, are also closely related to khasi foot_2 . khasi is written using the latin script. ## review of existing work while studying the existing work done with respect to machine translation system development involv-ing assamese-xx or xx-assamese languages we got higher number of works as compared to khasi-xx or xx-khasi languages. the works in mt involving assamese language were developed using both statistical and neural approaches, where the later became the contemporary technology to further the research in translation task. in laskar et al. (2020), the authors have reported about developing statistical machine translation (smt) and nmt baseline works for assamese and english bilingual translation system in both directions. they have used two-layer network consists of long short term memory (lstm) cell with 500 nodes in each layer and attention mechanism @xcite . before this baseline work on assamese nmt, there are few works @xcite , @xcite , @xcite , @xcite , @xcite , @xcite on mt development for assamese using smt. @xcite developed a rule based machine translation system for assamese and english using apertium @xcite . in the neural domain, there are works on assamese nmt development such as @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , @xcite . for mt systems involving khasi languages, we got one work (singh and vellintihun hujon, 2020) using smt (the authors also developed an nmt system in the same work). hujon et al. (2023a) used convolutional sequence to sequence learning for english-khasi neural machine translation. use of both supervised and unsupervised technique is reported in thabah et al. (2019) for khasi to english nmt. in nonghuloo et al. (2020), the authors have used attention mechanism to get good bleu score for english to khasi nmt system. from this study of existing work, we conclude that almost all work on khasi mt system development, emphasis is given to translating khasi to/from english and no attempt is made to couple the language with other non-english languages. to fill this gap, we attempt to develop a baseline system involving khasi and non-english language, assamese, in our case. ## data collection and generation the languages, for which we are attempting to create nmt system, do not have any parallel data. what we had at our disposal are two resources: 1) english-khasi parallel data with 24000 sentences from wmt23 (indicmt subtask) and 2) english to assamese nmt system (developed in house) with 19.48 bleu score for our own test set and 15.16 bleu score for flores-200 test set @xcite . we analysed the english-khasi data to check the sentence length for source and target language. the figure 1 shows the density histogram of the sentence length variation for english and khasi. we got extremely unbalanced data in the parallel corpus (maximum english sentence length of 103 and maximum khasi length of 1723). so, we filtered the data in two ways. firstly, we have selected sentences from the original corpus where sentence lengths were less than or equal to 200 (set 1 figure 2a ) and secondly, we made the threshold criteria to be sentence length less than or equal to 50 (set 2 figure 2b ). a few sentences that are filtered from set 1 are filled from the first available sentences of the validation set (provided with the english-khasi indicmt subtask of wmt23) to keep the total number at 24000. the english side of both the two sets are then translated to assamese using our in-house english-assamese nmt system (as mentioned above). now, we have two sets of synthetic parallel corpus for assamese and khasi. set 1 has 24000 sentences (figure 2c ) and set 2 has 18508 sentences (figure 2d ). we split the data into training, testing and validation sets. the details of train-test-valid split is shown in again, to generate the test data, we have translated the english side of the test set provided by english-khasi indicmt sub task of wmt23, to assamese with our nmt system, and then manually post edited and validated the same. the test set contains 1000 sentences. ## experimental setup for the purpose of evaluating the influence of sentence length, we have performed four experiments in assamese to khasi and khasi to assamese language directions (two in each directions). moreover, we have also performed phrase table injec tion experiment on all the sets, again having four different models. the details of the experiments along with the machine details are shown in the following subsections. ## result the same consistency in terms of better score is observed for all three evaluation metrics holds in the case of phrase augmentation too. as-@xmath0. from both ## conclusion firstly, we conclude that, with only synthetic data, it is possible to develop a baseline system for language pairs where no parallel corpus is available previously. secondly, it is observed that sentence length plays a crucial role in getting good translation score. to establish the fact we would like to direct our attention in the future to formulate optimal sentence length for other language pairs. and lastly, when the size of training corpus is very small, extracting phrase pair from an smt model trained on the same data set and augmenting the phrases to the training set can further enhance the translation quality in terms of automatic evaluation metrics such as bleu, chrf and ter. in this work, we have created a synthetic khasi-assamese corpus and test set for the same pair. along with that we also created a baseline nmt system for khasi-assamese language pair. this is the very first work which demonstrates the use of nmt for khasi and non-english language. we encourage other researcher to pair diverse languages to create nmt systems.