Ixamed’s submission description for WMT 20 Biomedical shared task: benefits and limitations of using terminologies for domain adaptation (2020)

Abstract: In this paper we describe the systems developed at Ixa for our participation in WMT20 Biomedical shared task in three language pairs, en-eu, en-es and es-en. When defining our approach, we have put the focus on making an efficient use of corpora recently compiled for training Machine Translation (MT) systems to translate Covid-19 related text, as well as reusing previously compiled corpora and developed systems for biomedical or clinical domain. Regarding the techniques used, we base on the findings from our previous works for translating clinical texts into Basque, making use of clinical terminology for adapting the MT systems to the clinical domain. However, after manually inspecting some of the outputs generated by our systems, for most of the submissions we end up using the system trained only with the basic corpus, since the systems including the clinical terminologies generated outputs shorter in length than the corresponding references. Thus, we present simple baselines for translating abstracts between English and Spanish (en/es); while for translating abstracts and terms from English into Basque (en-eu), we concatenate the best en-es system for each kind of text with our es-eu system. We present automatic evaluation results in terms of BLEU scores, and analyse the effect of including clinical terminology on the average sentence length of the generated outputs. Following the recent recommendations for a responsible use of GPUs for NLP research, we include an estimation of the generated CO2 emissions, based on the power consumed for training the MT systems.

## introduction the wmt20 biomedical shared task calls for developing systems for translating biomedical abstracts and terminologies between several languages. in our case, we participate in the task of translating biomedical terms and abstracts from english into basque (en-eu), as well as translating biomedical abstracts between english and spanish (en-es and es-en). for translating the test data from english into basque, we concatenate our best en-es system with our es-eu system, both for translating abstracts and terminologies. ## related work for translating biomedical texts from english into catalan, costa-jussá et al. (2018) use a pivoting or cascade approach, translating the texts first from english into spanish (en-es), and then from spanish into catalan (es-ca). this technique is useful when there are more bilingual in-domain sentences for each of the language pairs (en/es and es/ca) than for the desired source and target languages (en/ca). since there are low resources for en/eu biomedical domain, but we have access to many resources for en/es and es/eu in the biomedical or clinical domain, we follow the same approach for translating the test sets from english into basque (en-eu). since most of the available in-domain corpus is monolingual, we also make use of traditional backtranslation and forward translation techniques (sennrich et al., 2016). in our previous work for translating clinical texts between basque and spanish, we showed that including clinical terminologies directly into the training corpus was useful for domain adaptation when no bilingual in-domain sentences were available (soto et al., 2019a). as clinical terminologies, we refer to the automatic translation into basque of snomed ct (ihtsdo, 2014), which is considered the most comprehensive, multilingual clinical health care terminology collection in the world. in this work, we extend the number of clinical terminologies as part of the ongoing translation of snomed ct into basque (perez-de-viñaspre, 2017), and include the provided icd-10 resources plus other smaller terminology collections recently created for translating covid-19 related texts. ## resources for training our baseline en/es systems, we make use of the medline corpus provided by the organisers of the wmt20 biomedical shared task, as well as the recently compiled taus corona crisis corpus. @xcite for backtranslation (es-en) and forward translation (en-es), we use the english corpus prepared by sketch engine foot_1 , based on the covid-19 related corpus compiled for a recent kaggle competition (wang et al., 2020). as a final step, we include several clinical terminologies: 1) the icd-10 (en-eu) corpus provided by the organisers of the wmt20 biomedical shared task, adding the corresponding spanish counterparts; 2) terms obtained from the automatic translation into basque of snomed ct (perez-de-viñaspre, 2017), including terms up to 11 tokens; 3) a recent snomed ct interim release of covid-19 related terms foot_2 , manually translated into basque by a translator of the basque public health service (osakidetza); and 4) a collection of covid-19 related terms recently compiled by elhuyar foot_3 , including all the terms published until june 18 @xcite . for training our es-eu system, we use the aforementioned terminologies together with an out-ofdomain corpus formed mainly by news (etchegoyhen et al., 2016), previously applying a language identification tool foot_5 to exclude sentences where most of the terms are named entities like locations or person names. doing this, a bigger part of the vocabulary can be used to translate biomedical or clinical terms. furthermore, as in-domain corpus we use clinical notes in spanish coming from the hospital of galdakao-usansolo for forward translation and copying (currey et al., 2017). this corpus was compiled between 2008 and 2012. @xcite for the evaluation of en/es systems, we use khresmoi; foot_7 while for es-eu we use templates of clinical notes in basque written in the donostia hospital (joanes etxeberri saria v. edizioa, 2014), together with their manual translations into spanish made by a bilingual doctor. ## systems for en/es we develop 3 systems: 1) using only the bilingual in-domain corpus (medline + taus corona crisis corpus), 2) including the sketch engine covid-19 (en) corpus for backtranslation (esen) or forward translation (en-es), and 3) adding all the clinical terminologies from icd-10, snomed ct and elhuyar. for es-eu we train a unique system using the outof-domain corpus and the clinical terminologies, as well as the in-domain (es) corpus for forward translation and copying. for training the backtranslation (en-es) and forward translation (es-en) systems, we used the bilingual in-domain corpus (medline + taus corona crisis corpus); while for es-eu we used the out-ofdomain corpus and a reduced set of snomed ct terminologies, as used in soto et al. (2019b). all the systems are transformer (vaswani et al., 2017) models trained with opennmt (klein et al., 2017), using the recommended hyperparameters. @xcite when necessary, we halved the batch-size so that it could fit in 2 gpus, and accordingly doubled the value for gradient accumulation. we applied joint bpe-dropout (provilkov et al., 2020), with 32,000 merge operations for en/es and 90,000 for es-eu. as expected, backtranslation significantly improves the es-en results (around 5 bleu points); while the gains obtained with forward translation (en-es) are smaller (around 2 bleu points in the dev set and around 1 bleu point in the test set). however, we observe a slight decrease on bleu values when including the clinical terminologies on the training corpus for both es-en and en-es systems. for further analysing this, we calculate the average sentence length of the different evaluation corpora as translated by the different systems. we observe that, except for the dev set translated by the en-es systems, the lower sentence length is always obtained when using the system including the clinical terminologies. this is confirmed by a fast check of the outputs generated when translating the official test sets provided by the organisers, where we see that the sentences translated by these systems usually end before having translated all of the terms that appear in the input. overall, the sentence lengths of the generated translations are closer to the original sentence lengths when using the baseline systems; therefore, for en-es and es-en we submit as best systems the translations produced by the baseline systems, using only medline and taus corpora. regarding es-eu, in lang. system bleu es-en baseline (medline + taus) 40.65 baseline + backtranslation (bt) 40.71 baseline + bt + terminologies 39.96 en-es baseline (medline + taus) 41.71 baseline + forward translation (ft) 38.36 baseline + ft + terminologies 38.58 en-eu single (en-es) + ensemble (es-eu) 8.15 ensemble (en-es) + single (es-eu) 7.82 ensemble (en-es) + ensemble (es-eu) 8.84 comparing to the submissions made by other teams, our systems submitted for en/es obtain the lowest bleu scores among all the participants; while for en-eu our best run is the second among the best runs of each participant, only surpassed by the three runs submitted by elhuyar. finally, lang. system acc. bleu en-eu single (en-es) + ensemble (es-eu) 0.12 13.14 ensemble (en-es) + single (es-eu) 0.08 7.21 ensemble (en-es) + ensemble (es-eu) 0.13 14.81 surprisingly, the obtained automatic scores are much lower than the ones obtained by the rest of the participants (between 0.73 and 0.78 for accuracy, and approximately 71 to 74 bleu scores). however, the generated translations look quite sensible, so we expect the human evaluation will shed some light about the performance of our systems. ## measured power consumption and estimated co 2 emissions following the recommendations by strubell et al. (2019), we report the power consumed by our gpus when training the systems developed for this work, along with the estimated co 2 emissions. for calculating the training time, we use the time shown in the first and last lines of the log file generated while training the systems, including also the initial time for preparing the data, so the presented values constitute an upper bound of the actually consumed power. nonetheless, we have to point out that opennmt makes an efficient use of the power capabilities of the gpus, so we can say that the numbers shown here are an accurate estimation. ## conclusion and future work in this work, we have presented a simple proposal using previously compiled corpora from the biomedical or clinical domain, as well as clinical terminology included directly to the training corpora. apart from calculating bleu scores, we have also calculated the average sentence length of the generated translations for en/es systems, and observed that the systems including terminologies performed generally worse than the baseline systems. as future work, we plan to incorporate these clinical terminologies in a more efficient way (dinu et al., 2019; wang et al., 2019). for improving both training and evaluation, we'll also use bilingual clinical domain corpora being compiled now in collaboration with the basque public health service (osakidetza). furthermore, since we have observed that some of the translations generated by the es-eu systems remain in spanish, we'll study techniques to leverage in-domain monolingual data in basque like the one provided by the organisers from wikipedia. finally, we plan to keep reporting the consumed power and consequently generated co 2 emissions, probably making use of recently developed automatic tools (henderson et al., 2020) foot_10 .