Elhuyar submission to the Biomedical Translation Task 2020 on terminology and abstracts translation (2020)

Abstract: This article describes the systems submitted by Elhuyar to the 2020 Biomedical Translation Shared Task, specifically the systems presented in the subtasks of terminology translation for English-Basque and abstract translation for English-Basque and English-Spanish. In all cases a Transformer architecture was chosen and we studied different strategies to combine open domain data with biomedical domain data for building the training corpora. For the English-Basque pair, given the scarcity of parallel corpora in the biomedical domain, we set out to create domain training data in a synthetic way. The systems presented in the terminology and abstract translation subtasks for the English-Basque language pair ranked first in their respective tasks among four participants, achieving 0.78 accuracy for terminology translation and a BLEU of 0.1279 for the translation of abstracts. In the abstract translation task for the English-Spanish pair our team ranked second (BLEU=0.4498) in the case of OK sentences.

## introduction general purpose translation systems usually perform poorly where domain specific knowledge is required @xcite . therefore, it is essential to develop translation systems for specific domains. however, high quality parallel corpora for domain specific tasks are only available for a few major languages and obtaining in domain translated data for the majority of language pairs becomes a challenge. with such scarcity of resources, various domain adaptation techniques have shown promising results @xcite @xcite . the biomedical domain is of great interest for the application of machine translation. it is a sector of great importance in society, even more so after the advent of covid-19, where the handling of documentary information plays a major role. therefore, it is a scenario where machine translation can be of great help in order to facilitate the flow of information between different languages. we can find different works in the literature that address the task of developing nmt systems adapted to the biomedical domain @xcite @xcite . in contrast, few papers focus on language pairs where training data is scarce. this is precisely the case of the task of translation from english to basque. the work of @xcite is specially relevant in this case, which presents a clinical domain oriented system for basque-spanish based on rnn and transformer architectures and which does not require bilingual domain texts but only bilingual clinical terminology (snomed ct). our participation in the biomedical translation shared task addresses the translation of biomedical terminology from english to basque and the translation of biomedical abstracts from english to basque and from english to spanish. given the scarcity of real parallel biomedical domain corpora, our approach focused on different strategies to combine open domain data with in-domain biomedical data. in the case of the english to basque translations, back translation technique has been applied to generate synthetic examples from a real biomedical spanish-basque corpus and several monoligual in-domain basque corpora have been merged to the training data by copying target side examples to source. furthermore, we have considered finetuning previously available open domain systems in order to take advantage of the learnt general, open domain patterns. this article is structured as follows: the following section reviews the most relevant related works. section 3 describes the systems presented to the sub-tasks that include the english-basque and english-spanish pairs, as well as the results obtained in the experimentation. section 4 presents the results obtained in the official evaluation, and finally, we end the paper outlining the most relevant conclusions drawn from this work. ## related work most of the related work on domain adaptation focuses on using either in-domain monolingual corpora, synthetically generated corpora or small parallel corpora. regarding in-domain monoligual corpora, currey et al. (2017) analyze the benefits of augmenting available data by copying the target side monolingual data to source and training the system. results show significant gains in accuracy on named entities and words remaining identical in source and target languages. several studies show the effectiveness of generating synthetic parallel corpora for domain adaptation. @xcite use the back translation technique with target monolingual data to strengthen the decoder, zhang and zong (2016) make use of source side monolingual data and @xcite use both, source and target monolingual data to improve in-domain translations. finally, when in-domain parallel corpora is available, previous work focuses on mixed domain nmt systems by using both in-domain and out of domain data. @xcite propose to use control tags to mark in-domain sentences prior to concatenating multiple domain corpora. @xcite compare different methods for training a multi domain system, such as, concatenation, interactively training on different domains, selecting out of domain data close to the in domain data and ensembling different domain models. @xcite exploit the internal sentence embeddings to find sentences that are close to in-domain data from out of domain data. several works in the literature address the task of developing nmt systems adapted to the biomedical domain. @xcite apply transfer learning technique by training on a large, general domain corpus and finetuning a series of systems on different biomedical domains. they perform multi domain ensembling to further improve the results. @xcite iteratively apply transfer learning on various biomedical domains. regarding works dealing with basque the work of @xcite presents a clinical domain oriented system for basque-spanish based on rnn and transformer architectures, which does not require bilingual domain texts but bilingual clinical terminology (snomed ct). they also analyse different back translation techniques. aimed at translating clinical terminology into basque, we find the work of perez-de viñaspre (2017) which proposes a system for translating snomed ct into basque by combining lexical resources, transliteration of neoclassical terms, generation of nested terms and a domain-adapted rbmt system. ## experiments in this section we describe the experiments carried out when training the translation systems. we compare the results obtained by the systems on different open domain and in-domain test sets prior to selecting the best runs to submit for the biomedical translation task. we also detail the transformer architecture used and its parameters. ## official results for the english to basque abstract translation task we selected the pro test as the most representative one when choosing the best runs for submission. we assumed that this test set would be the closest to the official task test. the ehu test set was also a great indicator of how robust our system was for the biomedical domain. bleu scores were calculated using the multieval tool and tokenization as provided in moses. in the case of the terminology task, icd-10 test set is the official validation task and therefore the most representative one for choosing the best runs. for the evaluation of terminology we provide two metrics: (i) accuracy, by relying on strict matches (case insensitive) between ground truth ans predictions; and (ii) bleu score, as measured by the nltk module sentence bleu. our best submitted run has obtained the best score on the official task test, achieving 0.78 accuracy and a 0.7373 bleu score. when compared to other teams, all our submitted runs outperform all the runs, except for dcu mt's run2. it is worth mentioning that out baseline4 (run3) which was trained on task's english-basque data has outperformed almost all of the others teams' results. this highlights the improvements obtained by our systems over the baseline. finally, for the english to spanish abstract translation task wmt19 and wmt18 test sets were selected as reference for selecting the best runs. ## conclusions for the biomedical translation task 2020, we considered several strategies combining open domain and in-domain biomedical data. we have successfully applied transfer learning by fine-tuning a previously available open domain system with in-domain specific data. to tackle the scarcity of english-basque domain data, we have performed data augmentation by back translating real data. the systems submitted for the terminology and abstract translation tasks for the english-basque pair have ranked first on the official task test, achieving 0.78 accuracy for terminology translation and a bleu of 0.1279 for the translation of