Selecting Backtranslated Data from Multiple Sources for Improved Neural Machine Translation (2020)

Abstract: Machine translation (MT) has benefited from using synthetic training data originating from translating monolingual corpora, a technique known as backtranslation. Combining backtranslated data from different sources has led to better results than when using such data in isolation. In this work we analyse the impact that data translated with rule-based, phrase-based statistical and neural MT systems has on new MT systems. We use a real-world low-resource use-case (Basque-to-Spanish in the clinical domain) as well as a high-resource language pair (German-to-English) to test different scenarios with backtranslation and employ data selection to optimise the synthetic corpora. We exploit different data selection strategies in order to reduce the amount of data used, while at the same time maintaining high-quality MT systems. We further tune the data selection method by taking into account the quality of the MT systems used for backtranslation and lexical diversity of the resulting corpora. Our experiments show that incorporating backtranslated data from different sources can be beneficial, and that availing of data selection can yield improved performance.

## introduction the use of supplementary backtranslated text has led to improved results in several tasks such as automatic post-editing @xcite , machine translation (mt) @xcite , and quality estimation @xcite . backtranslated text is a translation of a monolingual corpus in the target language (l2) into the source language (l1) via an already existing mt system, so that the aligned monolingual corpus and its translation can form an l1-l2 parallel corpus. this corpus of synthetic parallel data can then be used for training, typically alongside authentic human-translated data. for mt, backtranslation has become a standard approach to improving the performance of systems when additional monolingual data in the target language is available. while @xcite show that any form of source-side data (even using dummy tokens on the source side) can improve mt performance, both the quality and quantity of the backtranslated data play a significant role in practice. accordingly, the choice of systems to be used for backtranslation is crucial. in @xcite , different combinations of backtranslated data originating from phrase-based statistical mt (pb-smt) and neural mt (nmt) were shown to have different impacts on the quality of mt systems. in this work we conduct a systematic study of the effects of backtranslated data from different sources, as well as how to optimally select subsets of this data taking into account the loss in quality and lexical richness when data is translated with different mt systems. that is, we aim to (i) provide a systematic analysis of backtranslated data from different sources; and (ii) to exploit a reduction in the amount of training data while maintaining high translation quality. to achieve these objectives we analyse backtranslated data from several mt systems and investigate multiple approaches to data selection for backtranslated data based on the feature decay algorithms (fda: bic ¸ici and yuret (2015); @xcite ) method. we exploit different ways of ranking the data and extracting parallel sentences; we also interleave quality evaluation and lexical diversity/richness information into the ranking process. while our empirical evaluation shows different results for the tested language pairs, this is the first work in this direction and lays a firm foundation for future research. nowadays, nmt @xcite @xcite , and in particular transformer @xcite achieves state-of-the-art results for many domains and language pairs. however, nmt requires a lot more data than other paradigms @xcite , which makes it harder to adapt to low-resource scenarios @xcite . using synthetic parallel data via backtranslation has been helpful in some low-resource usecases @xcite . for extreme cases with no bilingual parallel corpora, unsupervised mt can obtain reasonable results @xcite . however, its application to real low-resource scenarios is still a matter of study @xcite . in this work we are motivated by a real-world lowresource use-case, namely the translation of clinical texts from basque to spanish (eu-es). basque is a minority language, so most of the electronic health records (ehr) are written in spanish so that any doctor from the basque public health service can understand them. the development of a system for translating clinical texts from basque to spanish could allow basque-speaking doctors to write ehrs in basque, thus contributing to the normalisation of the language in specialised areas. we conduct our analysis in the scope of the eu-es translation of ehr use-case, as well as on a language pair and a data set that have been well studied in the literature -german to english (de-en) data used in the wmt biomedical translation shared task @xcite . as the eu-es medical data cannot be made publicly available due to privacy regulations, using the de-en data is a way to allow for the replicability of our work. ## related work one of the first papers comparing the performance of different systems for backtranslation was @xcite . the authors compared smt and nmt systems, obtaining similar results. closer to our work, @xcite also try rbmt, pb-smt and nmt systems for backtranslating ehrs from spanish into basque. however, both papers are limited to comparing the performance of systems trained with backtranslated data originating from a single source, without examining whether a combination might be more effective. more recently @xcite combined the outputs of pb-smt and nmt systems used for backtranslation, showing that the combination of synthetic data originating from different sources was useful in improving translation performance. in this work we extend these ideas by combining backtranslated data from rbmt, pb-smt, nmt (lstm) and nmt (transformer); in addition, we use fda to select sentences translated by different systems and analyse the impact of data selection of backtranslated data on the overall translation performance. regarding the use of dataselection techniques in conjunction with synthetic data, @xcite fine-tune nmt models with sentences selected from a backtranslated set, and chinea-rios et al. (2017) select monolingual source-side sentences to generate synthetic target strings to improve the translation model. while the most common approach to assessing the translation capabilities of a mt system is via evaluation scores such as bleu @xcite , ter @xcite , chrf @xcite , and meteor @xcite , recently research has begun to address another side of quality of translated text, namely lexical richness and diversity. in a recent paper, @xcite study the loss of lexical diversity and richness of the same corpora translated with pb-smt and nmt systems. @xcite investigate the problem for seen (during mt training) and unseen text using mt systems trained on the europarl corpus @xcite , with original (human-produced and translated) text as well as in a round-trip-translation setting. 1 in this work we calculate the same lexical diversity metrics as @xcite , and further use those metrics to improve the data selection process applied to backtranslated data. ## data selection for backtranslation from multiple sources fda @xcite is a data selection technique that retrieves sentences from a corpus based on the number of n-grams overlapping with those present in an indomain data set referred to as s seed . fda scores each candidate sentence s according to: (i) the number of n-grams that are shared with the seed s seed ; and (ii) the n-grams already present in a set l of selected sentences, as defined in (1): in the context of mt, fda has been shown to obtain better results than other methods for data selection @xcite . acordingly, in this work we too focus on fda, although our rescoring idea is more general and can be applied to other selection methods based on n-gram overlap. related work on quality and lexical diversity and richness of mt demonstrates that (i) regardless of the overall performance of an mt system (as measured by both automatic and human evaluation), in general machine-translated text is error-prone and cannot reach human quality @xcite ); and (ii) machine-translated text lacks the lexical richness and diversity of human-translated (or postedited) text @xcite . in its operation, fda compares two types of text -the seed and the candidate sentences -without taking into account the quality or the lexical diversity/richness of the candidate text. our hypothesis is that when selecting data from different sources, fda cannot account for the differences in quality and lexical diversity/richness of these texts, with the consequence that the selected set (l) is sub-optimal. we test our hypothesis by assessing the quality and lexical diversity/richness of the backtranslated data with the four different systems as well as with different selected subsets of training data. to tackle the problem of sub-optimal fdaselected datasets, we propose to rescore fda scores based on quality evaluation and lexical diversity/richness scores. 2 that is, for each sentence 2 we talk about "rescoring" as if we compare equations ( 1 ) and ( 2 ), the only difference is the rescoring produced by multiplying equation (1) (left part in equation ( 2 )) by the s bt i from a backtranslated corpus d bt i originating from the i th mt system, we factor in the quality expressed by the evaluation metrics, q(d bt i ) and the lexical diversity/richness expressed by the diversity metrics, d(d bt i ) as shown in (2): fda for mt, we use a devset as the seed. in our method we compute bleu and ter on the devset also used as a seed; mtld is computed on the backtranslated text, i.e. the synthetic source text. ## language pairs -challenges and objectives as a challenging low-resource scenario, we chose the translation of clinical texts from basque to spanish, for which there is no in-domain bilingual corpora. we make use of available ehrs in spanish coming from the hospital of galdakao-usansolo to create a synthetic parallel corpus via backtranslation. the galdakao-usansolo ehr corpus consists of 142,154 documents compiled between 2008 and 2012. after deduplication, we end up with a total of 2, 023, 811 sentences. 3 as a basis for training the mt systems for backtranslation, we use a bilingual out-of-domain corpus of 4.5m sentence pairs: 2.3m sentence pairs from the news domain @xcite , and 2.2m from administrative texts, web-crawling and specialised magazines. in order to adapt the systems to the clinical domain, we used a bilingual dictionary previously used for automatic clinical term generation in basque (perez-de-viñaspre, 2017), consisting of 151,111 terms in basque corresponding to 83,360 unique terms in spanish. to evaluate our eu-es systems, we use ehr templates in basque written with academic purposes (joanes etxeberri saria v. edizioa, 2014) together with their manual translations into spanish produced by a bilingual doctor. these 42 templates correspond to diverse specializations, and were written by doctors of the donostia hospital. after deduplication, we obtain 1,648 sentence pairs that are randomly divided into 824 sentence pairs for validation (devset) and 824 for testing. in order to test the generalisability of our idea, we use a well-researched language pair, german-to-english. as our out-of-domain corpus, we used the de-en parallel data provided in the wmt 2015 @xcite news translation task. the adaptation of systems to the medical domain with backtranslated data is performed using the ufal data collection. 4 we selected the following subsets: ecdc, emea, emea new crawl, muchmore, pattr medical and subtitles. the total amount of sentences was 2,555,138 which after deduplication was reduced to 2,335,892. after filtering misaligned and empty lines, foot_3 the resulting amount was 2,322,599 sentences. we used the en monolingual side. for development and test sets we used the cochrane and nhs 24 subsets from the himl 2017 set. 6 ## empirical evaluation via a set of experiments, we (i) investigate the differences in the backtranslated data originating from the four different mt systems and their impact on the performance of mt systems using this backtranslated data, and (ii) test our hypothesis as well as different approaches to rescoring the data selection algorithm. ## conclusions and future work we evaluated several approaches to data selection over the data backtranslated by rbmt, pb-smt, lstm and transformer systems for two language pairs (eu-es and de-en) from the clinical/biomedical domain. the former is a lowresource language pair, and the latter a well researched, high-resource language pair. furthermore, in terms of the two target languages, english is a morphologically less rich language than spanish, which creates a different setting again in which to evaluate our methodology. we use these two different use-cases to better understand both data selection and backtranslation. we show how the different fda data selection configurations tend to select different numbers of sentences coming from different systems, resulting in mt systems with different performance. under the assumption that fda's performance is hindered by the fact that the data originates from mt systems, and as such contains errors and is of lower lexical richness, we rescored the data selection scores for each sentence by a factor depending on the bleu, ter and mtld values of the system used to backtranslate it. by doing so, we managed to improve the results for the de-en system, while for eu-es we obtained similar performance to the other mt systems; this allows us to use just 25% of the data. further investigation is required to study under which conditions our proposed rescoring method is beneficial, but our experiments with both low-and high-resource language pairs suggest that if the systems used for backtranslation are poor, then this technique will be of little value; clearly this is closely related to the amount of resources available for the language pair under study. in the future, we plan to investigate ways to directly incorporate the rescoring metrics into the data selection process itself, so that penalising similar sentences can also be taken into account. we also aim to conduct a human evaluation of the translated sentences in order to obtain a better understanding of the effects of data selection and backtranslation on the overall quality. finally, we intend to analyse the effect of these measures in a wider range of language pairs and settings, in order to propose a more general solution.