Comparing and combining tagging with different decoding algorithms for back-translation in NMT : learnings from a low resource scenario (2022)

Abstract: Back-translation is a well established approach to improve the performance of Neural Machine Translation (NMT) systems when large monolingual corpora of the target language and domain are available. Recently, diverse approaches have been proposed to get better automatic evaluation results of NMT models using back-translation, including the use of sampling instead of beam search as decoding algorithm for creating the synthetic corpus. Alternatively, it has been proposed to append a tag to the back-translated corpus for helping the NMT system to distinguish the synthetic bilingual corpus from the authentic one. However, not all the combinations of the previous approaches have been tested, and thus it is not clear which is the best approach for developing a given NMT system. In this work, we empirically compare and combine existing techniques for back-translation in a real low resource setting: the translation of clinical notes from Basque into Spanish. Apart from automatically evaluating the MT systems, we ask bilingual healthcare workers to perform a human evaluation, and analyze the different synthetic corpora by measuring their lexical diversity (LD). For reproducibility and generalizability, we repeat our experiments for German to English translation using public data. The results suggest that in lower resource scenarios tagging only helps when using sampling for decoding, in contradiction with the previous literature using bigger corpora from the news domain. When fine-tuning with a few thousand bilingual in-domain sentences, one of our proposed method (tagged restricted sampling) obtains the best results both in terms of automatic and human evaluation. We will publish the code upon acceptance.

## introduction neural machine translation (nmt) @xcite is the state-of-the-art approach for developing machine translation (mt) systems. however, as nmt is based on artificial neural networks, its performance is dependent on big quantities of bilingual sentences, which are not available for all language pairs and domains. back-translation (bt) @xcite , based on the automatic translation of a corpus from the target language into the source language for augmenting the training data, has become a de facto standard for improving the performance of nmt models, provided that large monolingual corpora in the target language and domain are available. when generating a translation, considering that looking for all the possible output sentences is practically infeasible, mt systems have to implement an efficient technique for selecting the most probable sentence according to the distribution of the training data. typically, beam search @xcite is used for generating both the output sentences of nmt systems and the synthetic sentences produced by bt systems. @xcite proposed to use sampling for bt as one way to further improve the performance of nmt systems. specifically, their 'unrestricted sampling' foot_0 approach, consisting of randomly sampling from the output distribution, obtained the best results on average comparing to other decoding algorithms, including beam search. on the contrary, @xcite suggest that the improvement derived from using sampling for bt comes from the fact that the final nmt system can identify the synthetic corpus for having been generated by sampling instead of beam search, so they propose a simple alternative consisting of adding a tag to the corpus generated by the bt system using traditional beam search. they also tried to tag the output of the bt system using noising as proposed by @xcite , but they did not combine tagging with sampling. concurrent work by @xcite instead propose some variations to the sampling approach, consisting of disabling the label smoothing option when training the bt system, and restricting the sampling by setting a minimum value to the probability of the output sentences or limiting it to the top-k values. from these options, the last one obtained the best results, which we refer to as 'restricted sampling'. thus, we would have six options for generating the bt corpus, depending on which decoding algorithm is used, and whether tagging is used or not. from these combinations, the last two are proposed for the first time in this work: we compare these 6 methods both in terms of automatic evaluation of nmt systems, and lexical diversity (ld) of the synthetic corpora created by the bt systems. for mt automatic evaluation we use bleu @xcite , ter @xcite , chrf @xcite , and meteor @xcite ; while for lexical diversity we measure ttr @xcite ), yule's i @xcite and mtld @xcite . in the following, we briefly describe the lexical diversity metrics, for being less known. ttr, standing for type-token ratio, is the most common measure for lexical diversity. its value is obtained by dividing the number of typesdefined as the number of different words-by the total number of tokens or words in a given corpus. while easy to interpret, ttr is limited in the sense that their values differ significantly when changing the corpora size, thus it is only a valid metric for comparing lexical diversity of similar sized corpora. yule's i is the reversion of yule's k, or "characteristic constant", which represents the variability of the lexical frequency as the analysed text from the corpus under study gets bigger. yule's i and yule's k are thought to be less sensitive to changes in the corpora size. however, both ttr and yule's i are considered as better suited for small sized corpora. mtld or measure of textual, lexical diversity, sequentially measures the mean length of subsequent n-grams that have the same ttr value. as it is measured sequentially, it is less prone to changes in the values measured on different sized corpora, and it is considered as the most representative metric for measuring the lexical diversity of big corpora as the ones typically used in mt. as a complement to our mt and ld metrics, we add the results coming from a preliminary human evaluation done by a bilingual biomedical expert. according to these results, we select the best two systems for translating clinical reports from basque to spanish, and ask bilingual healthcare workers to post-edit the outputs of these systems, as well as the system trained in the opposite direction. finally, we report an estimation of the carbon footprint produced when developing our systems, which can be considered for deciding which approach to take in future works. ## related work apart from the works mentioned in the introduction proposing different methods for decoding or tagging the synthetic bt corpus @xcite @xcite , there is some other previous work on comparing different systems for bt. probably the most relevant work in this respect is the one that compares different techniques (i.e.: rule-based, statistical or neural mt) for generating the synthetic bt corpus. in this area, the work by @xcite firstly compared the use of statistical (smt) and neural (nmt) systems for bt, without observing significant differences. more similarly to our work, @xcite tried rule-based (rbmt), smt and nmt for bt ap-plied to the translation of clinical texts, obtaining better results with nmt, and specifically the transformer architecture @xcite . @xcite went one step further and not only compared the performance of different techniques for bt, but combined the synthetic corpora created by smt and nmt systems, probing that the combination of the outputs of both systems was useful. furthermore, @xcite compared and combined the outputs of rbmt, smt and nmt systems for bt, also analysing the lexical diversity of the generated corpora. they observed that the combination of all systems was in general better than using the output of only one system, and tried to improve the performance by applying data selection @xcite to the bt corpus, conditioned on the measured mt and ld metrics for each of the bt systems. regarding the use of tags for identifying the bt corpus, @xcite concluded that it was advisable to add a tag when the origin of the text was unknown, since systems using bt without a tag overfitted to the synthetic corpus, and even shown to be detrimental when used to translate text originally written in the source language. finally, our analysis of the lexical diversity of the bt data generated by different methods follows the work of @xcite , where the authors study the loss of lexical diversity of a given corpus after being translated with smt and nmt systems. therefore, in our work we measure the lexical diversity of the bt corpora according to the same metrics they calculate. ## material and methods we test the six methods presented in the introduction for a real use case: the translation of clinical notes from basque to spanish (eu-es). this work is part of an ongoing project that aims to implement an mt system in the basque public health service (osakidetza), so basque speaking healthcare workers can write their reports in basque without compromising the safety of their patients. 2 the first step in this project is the compilation of a basque/spanish (eu/es) parallel corpus of health records to be used for fine-tuning and evaluation, while previously collected spanish monolingual corpora will be used for bt. since these corpora are private, we reproduce our experiments in a similar setting for translating biomedical texts from german to english (de-en), using only publicly available data. for both language pairs, we preprocess our corpora by tokenizing and truecasing through moses tools. 3 further, we apply bpe @xcite for 90,000 (eu/es) and 40,000 (de/en) iterations. the number of bpe steps for eu/es was optimized in previous experiments, while the de/en one was taken from a reference system @xcite that will be described in section 3.2. for training all our systems, we use the transformer architecture as implemented in fairseq @xcite , with 6 encoder-decoder layers and an embedding size of 512. all the systems were trained for 30 epochs, except the es-eu system that was trained for 50 epochs due to applying the bpedropout @xcite regularization technique, as this setting obtained better results on preliminary experiments. in the future, we plan to do the same for the best performing eu-es systems. for de/en, we opt to use regular bpe for better reproducibility. in the following subsections, we describe the data used for each language pair. ## 75 83 we clearly observe that restricted sampling, which obtained the second best mt automatic scores but the lowest ld scores according to the most relevant mtld metric, gets significantly lower adequacy scores (75/100) in this preliminary human evaluation, while tagged restricted sampling and unrestricted sampling obtain the same number of totally correct translations (83/100). this confirms our intuition that, in the absence of a human evaluation, ld metrics can be used as a proxy to assess the mt quality of different systems trained with the same corpus. ## conclusions and future work in this work, we have empirically compared and combined different methods for bt applied to the mt of clinical texts. one of the new combinations tried in this work, tagged restricted sampling, obtained the best automatic scores according to all the metrics studied in the eu-es direction, confirmed by the hter and keystroke results from the human evaluation performed by bilingual healthcare workers. in the simulated low resource de-en scenario, traditional beam search still obtained the best mt results, followed by tagged beam search. this confirms the generalized agreement that sampling is only helpful when large monolingual data are available. moreover, we observe that tagging only helps when using sampling for decoding the bt systems, complementing previous work that proposed tagging the synthetic corpora as an alternative to the use of sampling. however, to drive more generalizable conclusions it would be necessary to try these methods on more diverse scenarios. considering the ld metrics, the decoding algorithm that obtained the best mt results in the eues scenario (restricted sampling) obtained one of the lowest mtld scores. in a preliminary human evaluation done by a bilingual biomedical expert to assess the 3 systems that obtained higher mt evaluation scores, restricted sampling obtained significantly worse results than unrestricted sampling, even that the latter obtained lower mt automatic scores. this is a sign that ld metrics can be used as a complement to the mt automatic evaluation scores for identifying the best performing systems. finally, we have estimated the carbon footprint derived from our experiments. we will consider these values to study possible ways of reducing or neutralizing our carbon footprint.