U o S Participation in the WMT 20 Translation of Biomedical Abstracts (2020)

Abstract: This paper describes the machine translation systems developed by the University of Sheffield (UoS) team for the biomedical translation shared task of WMT20. Our system is based on a Transformer model with TensorFlow Model Garden toolkit. We participated in ten translation directions for the English/Spanish, English/Portuguese, English/Russian, English/Italian, and English/French language pairs. To create our training data, we concatenated several parallel corpora, both from in-domain and out-of-domain sources.

## introduction in this paper, we present the system developed by the university of sheffield for the biomedical translation shared task in the fifth conference on machine translation (wmt20), which consists in translating scientific texts from the biological and health domain. our participation in this task considered the english/portuguese, english/spanish, english/russian, english/italian, and english/french language pairs with translations in both directions. for that matter, we developed a machine translation (mt) system based on neural machine translation (nmt), using google's tensorflow model garden. 1 2 related works previous participation in biomedical translation tasks include the works of @xcite which employed moses statistic machine translation (smt) to perform automatic translation integrated with a neural character-based recurrent neural network for model re-ranking and bilingual word embeddings for out of vocabulary (oov) resolution. given the 1000-best list of smt translations, the rnn performs a re-scoring and selects the translation with the highest score. the oov resolution module infers the word in the target language based on the bilingual word embedding trained on large monolingual corpora. their reported results show that both approaches can improve bleu scores, with the best results given by the combination of oov resolution and rnn re-ranking. similarly, @xcite also used the n-best output from moses as input to a reranking model, which is based on a neural network that can handle vocabularies of arbitrary size. more recently, tubay and costa-jussà (2018) employed multi-source language translation using romance languages to translate from spanish, french, and portuguese to english. they used data from scielo and medline abstracts to train a transformer model with individual languages to english and also with all languages concatenated to english. in the last two wmt biomedical translation challenges (wmt18 and wmt19) @xcite , the submissions that achieved the best bleu scores for the es/en and pt/en, in both directions @xcite @xcite @xcite , used the transformer architecture with enhancements such as handling of terminology during tokenization @xcite , multi-domain inference @xcite and exploitation of additional linguistic resources @xcite . ## resources in this section, we describe the language resources used to train both models. ## experimental settings in this section, we detail the pre-processing steps employed as well as the architecture of the transformer. ## results for the portuguese language, in both directions we achieved the best scores, with an en/pt bleu of 0.4744 and pt/en of 0.5334. the second team in both languages (unicamp dl) achieved scores of 0.4095 and 0.4988, respectively. as for the russian, french, and italian languages, our scores were not as competitive as the best systems, with the exception of fr/en, which we stood as 3 out of 5 teams. after carefully checking our training data, we found encoding issues with the different gathered data for those languages, especially with the encoding and tokenization of words containing apostrophes in french and italian, as well as the cyrillic kha. ## conclusions we presented the university of sheffield (uos) machine translation system for the biomedical translation shared task in wmt20. for our submission, we trained ten transformers nmt systems, employing different corpora for each language pair. in addition, for systems with english as target language, back-translation was used, and for the russian language, incremental training from patent abstracts was used. for model building, we included several corpora from biomedical and health domain, and from out-of-domain data that we considered to have similar textual structure, such as books and patents. prior training, we also pre-processed our corpora to ensure that we did not include any sentence from the released test set, which could produce biased models. regarding future work, we are planning on optimizing our systems by performing pre-selection of out-of-domain data, aiming at selecting only the most similar sentences to the in-domain data. in addition, we plan to explore the potential use of domain-specific decoding, as proposed in @xcite .