E du MT : Developing Machine Translation System for Educational Content in I ndian Languages (2021)

Abstract: In this paper, we explore various approaches to build Hindi to Bengali Neural Machine Translation (NMT) systems for the educational domain. Translation of educational content poses several challenges, such as unavailability of gold standard data for model building, extensive uses of domain-specific terms, as well as the presence of noise in the form of spontaneous speech as the corpus is prepared from subtitle data and noise due to the process of corpus creation through back-translation. We create an educational parallel corpus by crawling lecture subtitles and translating them into Hindi and Bengali using Google translate. We also create a clean parallel corpus by post-editing synthetic corpus via annotation and crowd-sourcing. We build NMT systems on the prepared corpus with domain adaptation objectives. We also explore data augmentation methods by automatically cleaning synthetic corpus and using it to further train the models. We experiment with combining domain adaptation objective with multilingual NMT. We report BLEU and TER scores of all the models on a manually created Hindi-Bengali educational testset. Our experiments show that the multilingual domain adaptation model outperforms all the other models by achieving 34.8 BLEU and 0.466 TER scores.

## introduction massive open online courses (moocs) have gained a lot of attention in recent years due to the availability of high-quality educational resources free of cost. in india, national programme on technology enhanced learning (nptel) foot_0 is one such initiative to promote online education. however, most of the content offered in english poses a problem for non-native english language speakers especially in a multilingual country like india. one potential solution to mitigate this problem is developing machine translation (mt) systems to translate contents from english to other indian languages. developing machine translation (mt) systems between two indian languages is more difficult than developing systems between english and indian languages due to the unavailability of the educational parallel corpus for indian languages. mt systems, especially current state-of-the-art neural machine translation (nmt) systems @xcite @xcite are data-hungry and requires a lot of training data @xcite . developing mt systems for the educational domain poses issues such as lack of data, translation of domain-specific terms, phrases, and mathematical expressions. since the dataset is prepared from the lecture subtitles and transcripts, it also contains noise in the form of spontaneous speech (e.g: 'umm', 'yes! good morning' etc.) and repetition of phrases (e.g: 'ok, well.. ok well, now we have to compute this value' etc.). due to these issues, building an mt system for the educational domain is a challenging task. in this paper, we focus on developing the nmt systems between two indian languages, namely @xmath0. we create a hindi-bengali educational corpus by crawling nptel lecture subtitles, transcripts that are in english and translating them into hindi and bengali. we create two types of educational parallel corpora, 'synthetic' and 'clean'. synthetic corpus is prepared from translating english data into hindi and bengali with the help of google translate 2 . clean corpus is prepared by manual post-editing of synthetic data via manual annotation and crowd-sourcing. we conduct experiments on prepared corpora with domain adapta-tion @xcite , objective. we experiment with denoising @xcite and automatic post-editing @xcite objectives to automatically clean the synthetic corpus which are further used to train the models. we also experimented on multilingual nmt @xcite using the english part of the corpus along with hindi and bengali. since there is no standard educational test set is available to test our models' performance, we manually create the test set by translating the hindi part of the english-hindi parallel corpus (ai domain) from adap-mt shared task (sharma et al., 2020) into bengali. we report bleu and ter score @xcite on the prepared test corpus foot_2 . the paper is organized as follows. in section 2, we briefly review a few of the notable works on building mt systems for educational content and domain adaptation, data augmentation methods in nmt. in section 3, we describe the corpus creation process. the experimental setup used to conduct domain adaptation and semi-synthetic data augmentation experiments are described in section 4 and section 5, respectively. the nmt model settings and experimental setup are described in section 6. results are described in section 7. finally, the work is concluded in section 8. ## related work building an nmt system for any domain requires a significant amount of data. in the educational domain, obtaining data is very challenging. most of the works in building mt systems for the educational domain is focused on creating corpora. @xcite have created educational corpora for 20 languages (20 monolingual and 190 parallel corpora) by crawling amara website foot_3 which is a community-driven web-based platform for editing and creating subtitles for videos. parallel corpus for european languages in educational domain have been created via crowd-sourcing kordoni et al. (2016); sosoni et al. (2018) ; @xcite . they built nmt systems on the prepared corpora and report that even a small amount of crowd-sourced translations can improve the translation quality. domain adaptation is a methodology to adapt models trained on out-of-domain data to in-domain data. @xcite proposed two methods for fine-tuning which do not need any modifications to standard nmt architecture. one method is to add domain tags (e.g: '<2domain>') and train the nmt model on the combined corpora from multiple domains. the second method is to fine-tune the model trained on out-of-domain data on the combination of in-domain and out-of-domain data. @xcite proposed three methods for domain adaptation. 'discriminative mixing' method uses a discriminator which is a fully connected layer, to predict the domain tag the current input sentence belongs to. the loss from discriminator and decoder is added and back-propagated which jointly optimizes the network. this makes the encoder encode domain-related features. 'adversarial discriminative mixing' method is the same as 'discriminative mixing' method except while backpropagating loss, the loss from discriminator is reversed by multiplying it with -1. this makes the encoder encode domain invariant features. 'target token mixing' does not use a discriminator network but simulates the discriminator by adding domain tags to the target sentence. improving the performance of nmt models with additional monolingual data is a common practice especially in low-resource settings. backtranslation @xcite is an effective approach to make use of the target monolingual data. @xcite conducted various experiments to generate synthetic source sentences from target monolingual data and used it to further train the models. they report that corrupting synthetic source sentences with noise and using that noisy source sentence instead of a clean synthetic source sentence, significantly improve the performance of the nmt models. multilingual nmt @xcite is another popular approach to improve the performance of nmt models for low resource language pairs by augmenting the low resource pairs with high resource language pairs and training a single nmt model. in this work, we build nmt models with domain adaptation objectives. we experiment with cleaning synthetic in-domain corpus with denoising auto-encoder @xcite and automatic post-editing @xcite objectives. the resulting data is augmented with created in-domain corpus and used to train nmt models. we also experiment with combining multilingual nmt and domain adaptation objectives. ## corpus creation we prepare the parallel corpus in educational domain by crawling lecture subtitles. specifically, we crawl the lecture subtitles from youtube and lecture transcripts from nptel courses foot_4 . the subtitles crawled from youtube are of smaller length compared to subtitles crawled from lecture transcripts. we crawl lectures on programming, data structures, algorithms, machine learning, and artificial intelligence. ## domain adaptation we consider both synthetic and clean hindi-bengali educational parallel corpus as in-domain 13 https://xsaras.com/ 14 please note that there is no overlap between annotators who post-edited the corpus and evaluators. we train two baseline models, namely 'outof-domain baseline' and 'in-domain baseline'. the out-of-domain baseline model is trained on samanantar corpus and the in-domain baseline model is trained on the prepared clean educational parallel corpus. we train two domain adaptation models by following fine-tuning @xcite method. specifically, we use the out-of-domain baseline model as the parent model. the parent model is fine-tuned with (i). clean educational parallel corpus (denoted as 'ft-clean') (ii). synthetic + clean educational parallel corpus (denoted as 'ft-both'). the reason to build two fine-tuned models is to check whether synthetic corpus is improving model performance or not. based on the results (ref we also experiment with adding domain tags foot_13 to source sentence @xcite (denoted as 'source token mixing') and target sentence @xcite (denoted as 'target token mixing'). using these methods, a single model can be trained on both out-of-domain and in-domain data at the same time. this will save time to train the model. in our case, since out-of-domain data size is very large compared to in-domain data, we oversample in-domain data to match the size of the out-ofdomain data. ## semi-synthetic data augmentation since most of our in-domain data is synthetic, we conduct experiments on automatic corpus cleaning. we experiment with two methods for automatic corpus cleaning, denoising auto-encoder @xcite and automatic post-editing (ape) @xcite . we conduct experiments on the bengali part of the corpus as it is our target language. we use synthetic-clean bengali sentence pairs from clean corpus 19 as our training corpus for corpus cleaning experiments. with the ape objective, we train an end-to-end nmt model with synthetic bengali sentences as input and clean bengali sentences as the target. @xcite show that when using back-translated @xcite data to train the nmt model, adding noise to input sentences improve model performance significantly. similarly, we create a noisy version of source sentences with two types of noise: (i). randomly dropping word with probability 0.1 17 since we created the educational corpus by translating english to hindi and bengali, we have 3-way parallel corpus involving hindi, bengali and english languages 18 we use ##2en, ##2bn tags to denote english and bengali respectively 19 since we created clean corpus from synthetic corpus, we have synthetic-clean sentence pairs. (ii). randomly swapping tokens with its neighboring token with probability 0.1 @xcite . we do not modify the target sentences. we also experiment by combining these two objectives and training a single model which can perform both denoising and automatic post-editing. after training, we use these models to generate clean bengali sentences from synthetic bengali sentences. we denote this as 'semi-synthetic' corpus since the source (hindi) is synthetic and the target (bengali) is automatically cleaned. the main reason to perform automatic corpus cleaning is to use the resulting clean corpus to improve the performance of the nmt model for the educational domain. to this extent, we repeat the experiment similar to 'ft-both' which is finetuning the model trained on samanantar corpus with educational corpus. however, now we use semi-synthetic corpus along with synthetic and clean corpora to fine-tune the model. 'ft-both + denoising' denotes the model fine-tuned with clean, synthetic corpora and semi-synthetic corpus obtained from the denoising experiment. 'ft-both + ape' denotes the model fine-tuned with clean, synthetic corpora and semi-synthetic corpus obtained from the ape experiment. similarly, 'ft-both + denoising + ape' denotes the model fine-tuned with clean, synthetic corpora and semisynthetic corpus obtained from the experiment combining denoising and ape objectives. the reason to combine the semi-synthetic data with clean and synthetic data is to provide the model with as much data as possible since in-domain data size is less compared to out-of-domain data. ## experimental setup all the models have trained on the transformer @xcite architecture. we use 6 layer encoder-decoder stacks with 8 attention heads. embedding and hidden sizes are set to 512, dropout @xcite rate is set to 0.1. the feed-forward layer consists of 2,048 cells. adam (kingma and ba, 2015) optimizer is used for training with 8,000 warm-up steps with an initial learning rate of 2. we use token-wise batching with batch size set to 2048 tokens. for fine-tuned models, the parent model is trained till convergence 20 and the child model is initialized with the last checkpoint from the parent model without resetting any hyper-parameters. all the models are trained till convergence and checkpoints are created after every 10,000 steps. all the checkpoints are averaged and considered the best parameters for the respective model. we use opennmt toolkit @xcite 21 to train the models. we tokenize the data into subwords with the unigram language model @xcite using sentencepiece @xcite implementation. for all the models except 'ft-multilingual', we learn subword rules on corpus obtained by concatenating in-domain and out-of-domain corpora. the size of subword vocabulary is 50k for both hindi and bengali. for the 'ft-multilingual' model, we learn joint subword vocabulary for hindi, bengali, and english by combining all the in-domain corpora and hindi-bengali out-of-domain corpora, and the size of joint subword vocabulary is 75k. at the time of decoding, the beam size is set to 5 with no length penalty. ## results and analysis we test all the models on the prepared hindi-bengali test corpus of size 2,630 and report bleu @xcite and ter @xcite scores, calculated with sacrebleu @xcite 22 . the models, source token mixing and target token mixing performance are less compared to fine-tuned models. despite a single model jointly trained for both in-domain and out-of-domain and can share information between both the domains, performance on in-domain data is not significant. both the models outperform baseline models but the target token mixing model achieves less bleu score (18.6) than the ft-clean model (21.5). similar to the ft-both-mixed model, adding outof-domain data is acting as noise which limits the performance of the model on in-domain data. the model fine-tuned with the multilingual educational corpus (ft-multilingual) achieve the highest bleu score of 34.8 and lowest ter score of 0.466 (higher bleu and lower ter scores are preferable) of all other models. we observe that adding more in-domain data is improving the model performance. in our case, we add english in-domain corpus (i.e. hindi-english and english-bengali) to the hindi-bengali corpus. since both hindi and bengali synthetic data were prepared from english data, adding english along with hindi-bengali helped the model to learn better representations for the hindi-bengali pair. this is evident from the experiments with bleu score of ft-multilingual model (34.8) improved by +1.2 points than ft-both model (33.6). similarly the ter score of the ft-multilingual model (0.466) improved by -0.016 24 points than ft-both model (0.482). 24 negative sign indicates the improvement as lower ter score is better. results from the semi-synthetic in-domain data augmentation models are interesting due to the reason that adding more in-domain data is not improving the performance. this observation is opposite of the observation from the ft-multilingual model where adding the english part of the parallel corpus is making the model outperform all other models. although the models, namely ft-both + denoising, ft-both + ape, and ft-both + denoising + ape are trained on in-domain corpus twice the size of actual in-domain corpus (since we add the semi-synthetic corpus to clean and synthetic corpus, the size of in-domain corpus become almost doubled) none of the models can outperform ft-both model (it only trained on clean and synthetic corpus). however, these three models outperform all other models except ft-both and ft-multilingual with ft-both + denoising model achieving the second-best ter score (0.481). we observe that the denoising objective is more effective than the ape objective for automatic corpus cleaning. we believe that if more synthetic-clean in-domain sentence pairs are available to train the denoising model, it will improve the quality of the semi-synthetic corpus which, in turn, improves the nmt model. we conduct a human evaluation on the output of our best model, namely ft-multilingual. we randomly choose 50 sentences from the test set and given to 4 evaluators 25 along with reference and output of the model and asked to evaluate based on adequacy and fluency on the scale of 1-5. the average adequacy and fluency scores are 3.5 and 3.85, respectively. based on the human evaluation, we conclude that the model can translate educational data with good adequacy and fluency. ## conclusion in this paper, we have explored the problem of building an nmt system in the educational domain for the hindi-bengali language pair. since there is no data available in the educational domain, we created the parallel corpus by extracting from lecture subtitles and transcripts and translating them into hindi and bengali. we also create a clean parallel corpus by post-editing the parallel corpus via crowd-sourcing as well as with the help of annotators. we trained neural machine translation models with domain adaptation objectives 25 these evaluators are the same who evaluated the quality of prepared clean in-domain corpus. by training models on publicly available samanantar hindi-bengali parallel corpus and fine-tuned with prepared educational data. we explored various methods to fine-tune the models such as mixed fine-tuning, source token mixing, and target token mixing. we experimented with data augmentation methods by automatically cleaning the synthetic in-domain corpus with denoising auto-encoder and automatic post-editing objectives. the resulting data is combined with prepared in-domain corpus and trained models. we also experimented with combining domain adaptation with multilingual nmt by training a model on samanantar hindi-bengali corpus and fine-tuned with multilingual in-domain corpus obtained by combining hindi-bengali, hindi-english, and english-bengali indomain corpora. since there is no standard test corpus is available, we created hindi-bengali educational test corpus through manual translation. we observed that the multilingual model outperformed all other models by achieving 34.8 bleu and 0.466 ter points. we also conducted a human analysis of the multilingual model by taking a sample of 50 random sentences evaluated based on adequacy and fluency metrics by 4 evaluators. the model achieved average adequacy and fluency scores of 3.5 and 3.85, respectively.