Training Deployable General Domain MT for a Low Resource Language Pair: E nglish- B angla (2018)

Abstract: A large percentage of the world’s population speaks a language of the Indian subcontinent, what we will call here Indic languages, comprising languages from both Indo-European (e.g., Hindi, Bangla, Gujarati, etc.) and Dravidian (e.g., Tamil, Telugu, Malayalam, etc.) families, upwards of 1.5 Billion people. A universal characteristic of Indic languages is their complex morphology, which, when combined with the general lack of sufficient quantities of high quality parallel data, can make developing machine translation (MT) for these languages difficult. In this paper, we describe our efforts towards developing general domain English–Bangla MT systems which are deployable to the Web. We initially developed and deployed SMT-based systems, but over time migrated to NMT-based systems. Our initial SMT-based systems had reasonably good BLEU scores, however, using NMT systems, we have gained significant improvement over SMT baselines. This is achieved using a number of ideas to boost the data store and counter data sparsity: crowd translation of intelligently selected monolingual data (throughput enhanced by an IME (Input Method Editor) designed specifically for QWERTY keyboard entry for Devanagari scripted languages), back-translation, different regularization techniques, dataset augmentation and early stopping.

## introduction today, machine translation (mt) is largely dominated by neural (nmt) and statistical mt (smt), with nmt, by far, becoming the most prevalent among the two @xcite . the performance of the corpus-based approaches to mt primarily depends on the availability of corpora to train them, specifically sufficient quantities of parallel data in a given language pair. this problem is exacerbated by nmt, which generally needs larger quantities of parallel data, and has stricter requirements as to the cleanliness of that data. unfortunately, large amounts of readily available parallel resources exist only for a small number of languages, e.g., opus @xcite and europarl @xcite , with very few sources of indic language data. while indian languages are widely spoken (in terms of native speakers), most of these languages have very little or no parallel resources available to build a general domain mt system @xcite . in the absence of readily available parallel corpora, comparable resources are often used to extract good quality parallel data from the web @xcite . in this direction also, indic languages have a very few comparable resources. a clear indication can be found by examining the number of wikipedia pages available for indic languages. we found only 57k pages are available for bangla (no indic language has more than 125k pages), while a large number of european languages have more than 1 million pages. furthermore, due to the usage of multiple fonts and encodings, a significant portion of the web data is not usable to extract useful parallel content. one of the major problems with training an nmt system on little data, especially when training an engine for general usage (i.e., not domain specific), is the problem of overfitting. deep neural networks have large parameter spaces and need ample amounts of data in order to generalize adequately; with small amounts of data they tend not to generalize well. we address this overfitting issue by learning the optimizer over a smaller number of steps. of course, adding more data always help, which is one of the benefits of synthetic data. in this paper, we describe our english (en)-bangla (bn) general purpose, production quality mt systems. bangla is the seventh most commonly spoken language in the world with an estimated reach of 215 million people in bangladesh and the indian subcontinent. first, we describe the smt-based system trained on approximately 1 million parallel sentences. bangla is a morphologically rich language, and as such, suffers from a high out-of-vocabulary (oov) rate in a low data scenario. we address the data sparsity issue through aggressive word segmentation technique. secondly, we build nmt models using the same parallel resources used for the smt systems. furthermore, we augmented a lot of synthetic training data @xcite generated using reverse translation engine to improve the nmt systems. the primary focus of this work is to develop general purpose mt systems for relatively low resource languages. the focuses of this work is summarized below. the rest of the paper is organized as follows. section 2 describes the data sets used to build the system. in section 3, we describe the smt and nmt models and their components. section 4 highlights the experimental setup and results. concluding remarks are made in section 5. ## data set the training data used to build our systems includes both true parallel data and synthetically generated parallel data using backtranslation @xcite . we use true parallel data to train both smt and nmt systems. however, the synthetic parallel data is used to train only the nmt systems. in this section we focus on the true parallel data and describe the generation of synthetic data in section 3.2. altogether, we have used 1m true parallel sentences along with larger synthetic data (approximately 2.8m and 8.2m for @xmath0, respectively). data from the web: often many web pages are available in multiple languages. some of these pages are sentence or paragraph aligned (lessnoisy) parallel data (eg. ted talks' transcription) and some articles are comparable or noisyparallel corpus in nature (eg. interlingually linked wikipedia documents). we have extracted several parallel and comparable web articles for bangla and english pair from the web. these articles for the most are not sentence aligned. once the potential parallel pages are extracted from the web, the sentence aligner is used to extract sentence aligned parallel text from the data. we extracted the data from the relevant file formats, and used a modified moore sentence aligner to align the data @xcite . crowd sourced data: we have used amazon's mechanical turk (mturk) for crowdsourcing the english to bangla parallel data creation task. this was primarily motivated from the work described in @xcite . in mturk, every task is divided into a set of human intelligence task (hit). in particular to our translation task, each hit consists of translating 10 sentences. the two key properties of our hits are reward amount ($0.50) and assignment duration (3 hours). furthermore, we incorporated automated quality checking into the hits for identifying incorrect entries made by turkers. this prunes some of the fraudulent entries and essentially reduces the manual approval time. the automatic check takes care of the following: one key issue with mturk is to identify a set of trusted users for the desired task as a lot of turkers provide bad data, e.g., by providing nonsense content, or most frequently, unedited mt'd content. we published 2 test hits (translate english into bangla and bangla into english) to find our trusted turkers based on the test hits. the turkers whose work has been approved manually were considered as trusted turkers. we had altogether a set of 24 trusted turkers from a total of 65 submissions. note, we integrated the indic language input tool (ilit) into the english into bengali hit interface so that the turkers can easily enter bangla text in the translation text box using a qwerty keyboard. due to the small size of the trusted crowd for bangla, it was time consuming to generate a large amount of parallel sentences using mturk. thus, we needed a careful selection process to choose the sentences which we wanted to translate to ensure maximum vocabulary saturation @xcite . we selected novel data based on the frequency distribution of the words in the existing parallel corpora. we ranked all the sentences in the un-translated source text based on the equation (1) and selected the top candidates (higher score) for manual translation. ## experiment and results first we conduct different experiments with the smt systems and compare the same with online (online-a) en-bn systems. the baseline smt experiments uses vanilla phrasal and treelet systems for @xmath1, respectively. furthermore, we conduct two different experiments using a word breaker (+wordbreak) and transliteration (+trans) in @xmath2. note, we have not used transliteration in @xmath3. we used bleu @xcite for automatic evaluation of our mt systems. the use of synthetic data also shows improvement over our final smt systems. we found an absolute improvement of 3.67 and 2.96 bleu points over the baseline phrasal @xmath4, respectively for webdunia and wmt testsets. similarly, we found an absolute improvement of 2.32 and 2.9 bleu points over the baseline in @xmath5, respectively for webdunia and wmt testsets. the use of bpe improves the performance with wmt testset, where there is little drop in bleu score with webdunia test set. this is due to the fact that the percentage of unknown word in wmt testset is much higher compared to web-dunia. finally, our system shows 0.9 and 2.51 absolute bleu point improvement over the online-a system in @xmath6. ## conclusion in this paper we presented en-bn smt and nmt systems, all of which were trained over a relatively small parallel corpus. the morphological richness of bangla exacerbates the problem of data sparsity, and we counter this problem through a variety of techniques and tools: developing a word breaker for bangla, generating synthetic parallel data, applying byte pair encoding (bpe) or morphological decomposition, and even crowd translating content based on vocabulary saturation data selection. additionally, we used early stopping to prevent overfitting. the mt systems and apis are publicly available in https://www.bing . com/translator. for future work, we plan to look into the integration of a word breaker into the nmt models (augmenting or replacing bpe). also, given the success we had with data selection, specifically, vocabulary saturation for the selection of content to manually translate, we plan to explore similar or related methods of data selection to improve the quality of synthetic data that we're translating (a la (junczys-dowmunt and birch, 2016), specifically applying @xcite ).