Effectively Aligning and Filtering Parallel Corpora under Sparse Data Conditions (2020)

Abstract: Parallel corpora are key to developing good machine translation systems. However, abundant parallel data are hard to come by, especially for languages with a low number of speakers. When rich morphology exacerbates the data sparsity problem, it is imperative to have accurate alignment and filtering methods that can help make the most of what is available by maximising the number of correctly translated segments in a corpus and minimising noise by removing incorrect translations and segments containing extraneous data. This paper sets out a research plan for improving alignment and filtering methods for parallel texts in low-resource settings. We propose an effective unsupervised alignment method to tackle the alignment problem. Moreover, we propose a strategy to supplement state-of-the-art models with automatically extracted information using basic NLP tools to effectively handle rich morphology.

## introduction machine translation (mt) quality has improved substantially with the advent of neural machine translation systems (nmt). however, while the quality gains over statistical machine translation (smt) systems can be large, in low-resource and domain mismatch settings they are significantly reduced @xcite . in recent years, unsupervised nmt trained only on monolingual corpora has attracted considerable attention, and has been proposed for scenarios where there is a lack of bilingual data @xcite . these methods have been shown to perform well for related language pairs (e.g. @xcite ), but as the languages differ more the unsupervised methods become less effective @xcite . @xcite show that supervised and semi-supervised baselines with only a small parallel corpus of 50k bilingual sentences consistently outperform the best unsupervised systems for a range of languages, similar and distant. they also show that unsupervised nmt is very sensitive to domain mismatch, which poses a problem to low-resource language pairs where it can be difficult to match the data domain on both sides. thus, it is evident that to achieve high quality mt, sentence aligned-texts in two or more languages are required. nmt systems have been shown to be sensitive to noise in the training data @xcite , where noise is defined as segments that decrease output quality of systems trained on the data. it is, therefore, important to be able to accurately align multilingual texts and precisely filter out misalignments and bad translations that adversely affect performance. in the study, conducted on the impact of various types of noise on mt quality, untranslated and misaligned segments had the most detrimental effect. misaligned segments were by far the most prevalent type of noise in the paracrawl foot_0 parallel corpus they used, twice as common as accepted segments. however, misalignments vary; a segment can have one extraneous word, it can have twice the content its counterpart has, or anything in between. it can be very useful to understand the intricacies of the effects different types and levels of noise have, why it is important not to have noise and whether some kinds of noise are more acceptable than others. this leads us to our first research question: rq1: how do different kinds of misalignments in a parallel corpus affect translation quality of an mt (smt or nmt) system trained on that corpus? if we can measure the effects of various misalignments, it could help us construct more effective methods to filter parallel corpora for mt. as the usefulness of parallel corpora for mt was first becoming apparent, harris (1988) pointed out that aligning such texts was a serious problem. moreover, collecting multilingual texts is expensive and time-consuming, and for some languages it can be hard to obtain access to even small amount of texts. thus, we need to be able to make the most out of what is available. we describe a method using bleualign (sennrich and volk, 2011) and monoses @xcite , an unsupervised smt system, to align parallel corpora using only monolingual texts for training. the proposed method is language pairindependent and only assumes unaligned bitexts and monolingual corpora for both languages. it is the first step towards answering our second research question: rq2: how can we best build useful parallel corpora from bilingual texts, having no other resources but monolingual corpora? in morphological typology, languages can be classified as analytic or synthetic (see e.g. haspelmath and sims (2013), @xcite ). analytic languages primarily rely on word order and auxiliary words to convey meaning, while synthetic languages use inflection. "morphologically rich" languages are synthetic languages which commonly have a large number of different surface forms for any given lexeme. this can lead to a high rate of out-of-vocabulary (oov) words, a data sparsity problem that machine learning algorithms struggle with. icelandic is a synthetic language with relatively few native speakers (approx. 350,000) where data sparsity problems are prevalent in most nlp tasks. in our work, we will focus on building a parallel corpus for the english-icelandic language pair and confronting the issues that arise when working with a less-resourced and morphologically rich language. when doing sentence alignment and filtering noise from parallel corpora, the sparsity problem caused by rich morphology leads to lower confidence scores for segment pairs resulting in lower classification accuracy, and thus smaller or less accurate parallel corpora. when parice (barkarson and @xcite , an english-icelandic parallel corpus was compiled, the filtering process resulted in an estimated 20% reduction in corpus size. out of what remained, about 5% was faulty (see section 3). we will work with the same data with the goal of minimising these numbers. this leads us to the third and last research question this research proposal centres around: rq3: how can we filter parallel corpora to minimize noise, and still lose little or no useful data from the original texts? our approach to try to answer these questions is to experiment with common and recent methods from the alignment and filtering literature. we will build a toolset that can employ various known methods and compare and contrast them. we will investigate how word embeddings, a lemmatizer, a part-of-speech (pos) tagger or a parser can help tackle the data sparsity problem, and which known methods benefit most from them. evaluation data sets will be created for the purposes of the project and the methods evaluated according to a set of evaluation metrics. finally, we will train and evaluate our system on a different language pair with comparable issues. ## related work filtering parallel data is the task of removing incorrect translations, noise and otherwise faulty data from a set of two (or more) aligned texts. alignment is the task of finding target segments with a corresponding meaning to that of source segments in multilingual texts. while these may seem to be different tasks, the same methods may apply partly to both problems. filtering is often done by scoring sentences and removing the lowest-scoring ones, whereas in alignment the highest-scoring sentences can be used as anchors: elements in the data that can reliably be aligned and thus direct further processing. in the next subsections, we describe alignment and filtering methods used in prior work. ## experimental framework the continuum of morphologically rich languages is quite diverse with the one end of the continuum being agglutinative languages, that primarily rely on discrete particles for inflection, and the other being fusional languages, which tend to use a single inflectional morpheme to denote multiple features. while it may be worthwhile to investigate if the same unsupervised methods work across different language categories, it can be expected that if further processing is needed, different approaches have to be taken. decompounding @xcite may be more useful for agglutinative languages to tackle the oov problem, and for many fusional languages internal change and suppletion call for different approaches. in our study we focus on fusional languages. english is primarily an analytic language and icelandic a fusional language with moderately rich morphology. we will be using the english-icelandic language pair as a test case. ## research plan our first goal is to set up an unsupervised pipeline for aligning parallel texts. while this is the first step in tackling rq2, it is also necessary to devise a method to answer rq1. we will outline how we seek to answer these questions, as well as rq3. a secondary goal is to investigate methods to improve upon the unsupervised pipeline by exploring how basic nlp tools can help us deal with the data sparsity problem inherent to many morphologically rich languages. in the following subsections we describe how we intend to research these questions. ## summary we have given an overview of the literature on sentence alignment and parallel corpus filtering. we outlined challenges associated with implementing these methods for low-resource and morphologically rich languages and proposed initial experiments to tackle these challenges. the motivation for this research is to improve the quality of machine translations by making better use of and increasing the quality of parallel training data, especially in regard to sparse data scenarios. an unsupervised method that effectively aligns bilingual texts will lower the barrier for building high-quality mt systems for low-resource languages and our first results suggest that it may also play a role in improving mt for morphologically rich languages.