Findings of the WMT 2020 Biomedical Translation Shared Task: B asque, I talian and R ussian as New Additional Languages (2020)

Abstract: Machine translation of scientific abstracts and terminologies has the potential to support health professionals and biomedical researchers in some of their activities. In the fifth edition of the WMT Biomedical Task, we addressed a total of eight language pairs. Five language pairs were previously addressed in past editions of the shared task, namely, English/German, English/French, English/Spanish, English/Portuguese, and English/Chinese. Three additional languages pairs were also introduced this year: English/Russian, English/Italian, and English/Basque. The task addressed the evaluation of both scientific abstracts (all language pairs) and terminologies (English/Basque only). We received submissions from a total of 20 teams. For recurring language pairs, we observed an improvement in the translations in terms of automatic scores and qualitative evaluations, compared to previous years.

## introduction automatic translation aims to alleviate the language barrier by providing access to information for readers not familiar with the original language used to write documents. access to accurate biomedical information is specifically critical and machine translation (mt) can contribute to making health information available to health professionals and the general public in their own language. it can also contribute to biomedical research by assisting with the writing of research reports in english. in addition, machine translation can provide the opportunity to enhance the use of natural language processing (nlp) tools and methods for low-resource languages by the development of resources through translation or by making tools available through text translation into resource rich languages. herein, we describe the fifth edition of the wmt biomedical task, 1 which aims to evaluate the auto-matic translation of a variety of biomedical texts. the first edition of the task @xcite ) focused on biomedical scientific abstracts in three language pairs. the second edition of the task offered ten language pairs and addressed scientific abstracts as well as patient-oriented health information @xcite . the third edition of the task offered six language pairs and addressed scientific abstracts @xcite . the fourth edition of the task offered ten language pairs. it addressed scientific abstracts and introduced the task of terminology translation @xcite . this year's edition of the task continues to address the translation of scientific abstracts and terminologies. it builds on previous tasks by offering a large range of training and test sets to support participants' systems. the following language pairs are addressed this year: this article is structured as follows: section 2 presents the details of the generation of our training and test sets, for both the scientific abstracts and the terminology, as well as manual validation of the quality of the test sets. section 3 describes our baseline systems, which are used as comparison in the automatic evaluation. we list all teams that participated in our task in section 4, as well as details of the methods behind their systems and the in-domain and out-of-domain data that was used. the results of the automatic evaluation based on the bleu and chrf scores are presented in section 5, while the ones for the manual evaluation are presented in section 6. finally, we discuss various topics related to the shared task in section 7. ## training and test data we provided training data of medline abstracts for it/en and ru/en, since training data for some of the other languages was already available from previous years. as for the tests sets, we released test sets for scientific abstracts and for terminologies, as summarized below: ## baselines we provided our baseline systems for all language pairs in the scientific abstracts translation subtask. there were two categories of baseline: for en/zh, en/fr, en/de, en/pt and en/es the models used for each direction were transformers @xcite trained by us using mariannmt @xcite with the following settings: joint bpe of 40,000, beam size 16. these parameters were chosen by tuning on a single direction of a single language pair: english to german. each of the 10 models were trained for up to two days. the training was stopped when there were no improvements on the validation dataset for more than 10 epochs, as measured through cross-validation score. the corpora we used to train the models were the same as last year -when we had baselines generated using rnn-based sequence2sequence models: the ufal medical corpus (ufa) without the "subtitles" subset, and as validation we again used khreshmoi @xcite . for en/it and en/ru and en/eu we used the helsinki-nlp/opus-mt-src-trg models (tiedemann and thottingal, 2020) included in the huggingface transformers library foot_13 , trained with mari-annmt on the entirety of the opus corpora (tiedemann, 2012). these models are not uniformly good; they performed very well for italian, but fairly poor for russian and basque. discussion. it is interesting that the models for english to/from italian performed so well in the biomedical task, as they were trained on generic text, not targeting the biomedical domain. it is interesting in general to what extent models that excel on generic text (e.g. news) perform well on the biomedical texts as well. ## teams and systems this year, 22 teams submitted a total of 151 runs. two teams withdrew after submitting their runs. the remaining teams were from china (7 teams), spain (3 teams), france (2 teams), the united kingdom (2 teams), armenia (1 team), australia (1 team), brazil (1 team), india (1 team), ireland (1 team) and pakistan (1 team). during the automatic evaluation, we observed that some teams obtained extremely high bleu scores, which were close to 0.9. those teams had trained their systems on the medline database, and the training data potentially included our test sets. unfortunately, as opposed to previous years, we forgot to inform participants on our website that this practice was not allowed. therefore, we offered the opportunity for these teams to re-submit their runs, but without training on medline. the wei-bot team was the only one to submit new runs. in an effort to increase the level of detail in the system description and the comparability between systems, we asked participants to fill in a survey with key information regarding the translation method used, as well as the in-domain and general datasets used for training. the survey comprised 14 questions covering the translation methods and corpora used. teams indicated their primary submission, which was considered for manual evaluation. on average, submission time for one language pair was 6 minutes and 28 seconds (median: 3 minutes and 35 seconds). all teams used transformer-based neural machine translation (except for team tramecat, who used sequence2sequence) and mostly relied on existing implementations: 19 teams submitted runs using available libraries, one team submitted runs using a mix of libraries and inhouse implementations, one team submitted runs exclusively relying on their own implementation of nmt. teams often used the same setup for a range of language pairs. for in-domain data, teams used the training data distributed by us and many of the sources described in @xcite . tables 7 and 8 provide details of the in-domain data used by the teams. for relevant language pairs, parallel data from other wmt tracks (e.g., news task) was used. interestingly, some teams used similarity measures based on biomedical corpora to extract additional biomedical sentences from out-of-domain corpora. out-of-domain data was also used in the form of pre-trained base models. ## automatic evaluation following @xcite , we used chrf @xcite as well as bleu @xcite as automatic metrics. chrf scores are obtained using the nltk implementation. 19 ## manual evaluation we manually validated a sample for each primary run in order to compare the performance between teams as well as to the reference translations. in this section we present details of the evaluation and results that we obtained. ## discussion in this section we present insights from the automatic and manual validations. we also reflect on the new processes introduced this year in the workflow of the task. ## conclusions we presented the findings of the fifth edition of the wmt biomedical task. this edition addressed three new languages and test sets that included scientific abstracts and terminologies. we explored news ways of improving our tests and carried out (as in previous editions of the task) both an automatic and a manual validation. results confirmed the improvements of the runs and for some language pairs, suggested that some runs were on a par with or superior to the reference translations.