Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for B engali- E nglish Machine Translation (2020)

Abstract: Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of 2.75 million sentence pairs, more than 2 million of which were not available before. Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. We also evaluate on a new test set of 1000 pairs made with extensive quality control. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status. To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We believe our study will pave the way for future research on Bengali-English machine translation as well as other low-resource languages. Our data and code are available at https://github.com/csebuetnlp/banglanmt .

## introduction recent advances in deep learning @xcite @xcite have aided in the development of neural machine translation (nmt) models to achieve state-of-the-art results in several language pairs. but a large number of high-quality sentence pairs must be fed into * these authors contributed equally to this work. with the above backdrop, in this work, we develop a customized sentence segmenter for bengali language while keeping uniformity with the english side segmentation. we experimentally show that better sentence segmentation that maintains homogeneity on both sides results in better alignments. we further empirically show that the choice of sentence aligner plays a significant role in the quantity of parallel sentences extracted from document pairs. in particular, we study three aligners and show that combining their results, which we name 'aligner ensembling', increases recall. we introduce 'batch filtering', a fast and effective method for filtering out incorrect alignments. using our new segmenter, aligner ensemble, and batch filter, we collect a total of 2.75 million high-quality parallel sentences from a wide variety of domains, more than 2 million of which were not previously available. training our corpus on nmt models, we outperform previous approaches to bengali-english machine translation by more than 9 bleu @xcite points and also show competitive performance with automatic translators. we also prepare a new test corpus containing 1000 pairs made with extensive manual and automated quality checks. furthermore, we perform an ablation study to validate the soundness of our design choices. we release all our tools, datasets, and models for public use. to the best of our knowledge, this is the first ever large scale study on machine translation for bengali-english pair. we believe that the insights brought to light through our work may give new life to bengali-english mt that suffered so far for being low in resources. we also believe that our findings will also help design more efficient methods for other low-resource languages. ## sentence segmentation proper sentence segmentation is an essential prerequisite for sentence aligners to produce coherent alignments. however, segmenting a text into sentences is not a trivial task, since the end-of-sentence punctuation marks are ambiguous. for example, in english, the end-of-sentence period, abbreviations, ellipsis, decimal point, etc. use the same symbol (.). since either side of a document pair can contain bengali/english/foreign text, we need a sentence segmenter to produce consistent segmentation in a language-independent manner. ## training data and batch filtering we categorize our training data into two sections: ## evaluation data a major challenge for low-resource languages is the unavailability of reliable evaluation benchmarks that are publicly available. after exhaustive searching, we found two decent test sets and developed one ourselves. they are mentioned below: sipc: @xcite used crowdsourcing to build a collection of parallel corpora between english and six indian languages, including bengali. although they are not translated by experts and have issues for many sentences (e.g., all capital letters on english side, erroneous translations, punctuation incoherence between bn and en side, presence of foreign texts), they provide four english translations for each bengali sentence, making it an ideal test-bed for evaluation using multiple references. we only evaluated the performance of @xmath0. @xcite : despite having many spelling errors, incorrect translations, too short (less than 50 characters) and too long sentences (more than 500 characters), due to its balanced nature having sentences from a variety of domains, we used it for our evaluation. ## related works the first initiative towards machine translation for bengali dates back to the 90s. @xcite developed anglabharti, a rule-based translation system from english to multiple indian languages, including bengali. asaduzzaman and ali (2003); dasgupta et al. (2004) conducted extensive syntactic analyses to write rules for constructing bengali parse trees and designed algorithms to transfer between bengali and english parse trees. subsequently, saha and bandyopadhyay (2005) reported an example-based machine translation approach for translating news headlines using a knowledge base. naskar and bandyopadhyay @xcite described a hybrid between rule-based and example-based translation approaches; here terminals would end at phrases that would then be looked up in the knowledge base. the improved translation quality of phrase-based statistical machine translation (smt) @xcite and the wide availability of toolkits thereof @xcite ) created an increased interest in smt for bangali-english. as smt was more data-driven, specialized techniques were integrated to account for the low amount of parallel data for bengali-english. among many, @xcite although nmt is currently being hailed as the state-of-the-art, very few works have been done on nmt for the bengali-english pair. @xcite trained a deployable general domain nmt model for bengali-english using sentences aligned from comparable corpora. they combated the inadequacy of training examples by data augmentation using back-translation @xcite . hasan et al. (2019); mumin et al. (2019a) also showed with limited parallel data available on the web that nmt provided improved translation for bengali-english pair. ## conclusion and future works in this work, we developed a custom sentence segmenter for bengali, showed that aligner ensembling with batch filtering provides better performance than single sentence aligners, collected a total of 2.75 million high-quality parallel sentences for bengali-english from multiple sources, trained nmt models that outperformed previous results, and prepared a new test set; thus elevating bengali from its low-resource status. in future, we plan to design segmentation-agnostic aligners or aligners that can jointly segment and align sentences. we want to experiment more with the laser toolkit: we used laser out-of-the-box, we want to train it with our data, and modify the model architecture to improve it further. laser fails to identify one-tomany/many-to-one sentence alignments, we want to address this. we would also like to experiment with bert @xcite embeddings for similarity search. furthermore, we wish to explore semi-supervised and unsupervised approaches to leverage monolingual data and explore multilingual machine translation for low-resource indic languages.