Pretrained Language Models and Backtranslation for E nglish- B asque Biomedical Neural Machine Translation (2020)

Abstract: This paper describes the machine translation systems proposed by the University of Technology Sydney Natural Language Processing (UTS_NLP) team for the WMT20 English-Basque biomedical translation tasks. Due to the limited parallel corpora available, we have proposed to train a BERT-fused NMT model that leverages the use of pretrained language models. Furthermore, we have augmented the training corpus by backtranslating monolingual data. Our experiments show that NMT models in low-resource scenarios can benefit from combining these two training techniques, with improvements of up to 6.16 BLEU percentual points in the case of biomedical abstract translations.

## introduction nowadays, most of the literature and scientific terminology produced in the biomedical field is in english, which limits the access to this information by non-english speaking researchers, doctors and patients. thus, it would be very useful to avail of machine translation systems that can effectively translate this information into other languages, so that more people can be able to access it and benefit from it. however, many of the world languages lack sufficient parallel corpora to properly train machine translation systems in this domain. state-ofthe-art neural machine translation (nmt) models @xcite suffer from overfitting when trained on insufficient data, and thus fail to generate accurate translations @xcite . in this paper we address this problem for a lowresource language, basque. we have taken part in the wmt20 biomedical translation challenge, which has released two interesting shared tasks involving this language, namely, the english-to-basque translation of biomedical article abstracts and the english-to-basque translation of medical terminology. in order to overcome the issue of having limited supervised training data, we have decided to apply two promising ideas proposed in the literature. first, we have applied transfer learning by training a bert-fused nmt model @xcite ) that uses source-language contextual embeddings inferred by a pretrained language model (lm) as additional input features, both in the encoder and in the decoder. second, we have augmented the training corpus using backtranslation @xcite . for this, a bert-fused nmt model has been trained in the opposite translation direction (@xmath0.g. wikipedia, medical texts). the experiments have shown that an nmt baseline can greatly benefit from combining these training techniques. the three best performing systems in both tasks (terminology and abstracts) have been submitted to the wmt20 biomedical translation shared task under the uts nlp team name. ## resources all the experiments have been carried out using only the parallel and monolingual training data recommended by the organisers on the shared task website. ## training and hyperparameter tuning we have trained a bert-fused nmt model @xcite with the open-source code provided by the authors foot_1 , which is built on top of fairseq foot_2 . following the authors recommendation, as a warmup step, first a standard transformer-based @xcite nmt model has been trained over the training data. then, this model has been used as both a baseline and to initialize the weights that the bert-fused nmt model has in common. we have used the transformer iwslt de en architecture as the nmt model, which consists of a 6-layer transformer network as the encoder and the decoder, with the embedding dimension set to 512 and the hidden layer dimension to 1024. additionally, we have used the following training hyperparameters: dropout 0.1, label-smoothing 0.1, inverse sqrt learning scheduler, warmup updates 4, 000, warmup initialization learning rate 1e -7 , minimum learning rate 1e -9 , weight decay 0.0001, bert encoder dropout 0.5 and the adam optimizer @xcite . the learning rate [0.0002, 0.00002] and the number of tokens per batch [1024, 4048] have been tuned using the validation set. all the datasets have been lowercased and tokenized using the moses tokenizer. additionally, we have learned subword units using byte pair encoding (bpe) @xcite with 10, 000 merge operations in order to reduce the vocabulary size and handle unknown words. in the terminology translation task we have only used the in-domain icd-10 code description data to train our models, because adding additional ood parallel data or backtranslated data was degrading the performance of the model. this is probably likely due to the specific and structured language used in the code descriptions, which is very different from the rest of the available texts. the baseline nmt was warmed up for 50 epochs and the best model over the validation was selected. then, the bert-fused models was tuned for 10 more epochs. in the abstract translation task, due to the fact that in-domain parallel data were not available, we have explored training the model with different combinations of the ood parallel data, the icd-10 training data and the backtranslated data. the icd-10 data and the backtranslated biomedical data have been upsampled x5 and x10, respectively. in this task, the baseline nmt was warmed up for 30 epochs, as the training data are much larger (longer training times) and because we have seen no noticeable improvement after the 30th epoch. like in the previous task, the bert-fused models have been tuned for another 10 epochs over the best baseline model. evaluation of the models has beeen carried out using the standard bleu metric @xcite . in the case of the terminology translation task, we have also used a case-insensitive strict accuracy metric, in which an icd-10 code description is considered correct only if it is a complete string match with the reference (no partial scores). ## conclusion this work has described the translation systems submitted by the uts nlp team to the wmt20 biomedical translation shared task. the proposed systems are bert-fused nmt models trained on a combination of in-domain parallel data, out-ofdomain parallel data and backtranslations of monolingual data. the experiments have shown that combining pretrained bert lms and backtranslations during training has contributed to achieve considerable accuracy improvements with respect to a standard transformer-based nmt model trained only on the parallel data. nevertheless, the official test results have shown that the performance of the systems can significantly drop if the translation domain is different. therefore, there is still significant work to do to improve the domain adaption of these models.