Findings of the WMT 2021 Biomedical Translation Shared Task: Summaries of Animal Experiments as New Test Set (2021)

Abstract: In the sixth edition of the WMT Biomedical Task, we addressed a total of eight language pairs, namely English/German, English/French, English/Spanish, English/Portuguese, English/Chinese, English/Russian, English/Italian, and English/Basque. Further, our tests were composed of three types of textual test sets. New to this year, we released a test set of summaries of animal experiments, in addition to the test sets of scientific abstracts and terminologies. We received a total of 107 submissions from 15 teams from 6 countries.

## introduction machine translation (mt) is the automatic translation of textual resources from one language to another. it is an important component in many applications and natural language processing (nlp) pipelines in the clinical and biomedical domains. on the one hand, some resources, such as specific biomedical terminologies, are only available for a limited number of languages. english is especially well covered in the unified medical language system (umls) @xcite while other languages are not @xcite . on the other hand, there are many publications written in languages other than english and are therefore inaccessible to researchers who cannot read those languages. this context has been the overarching goal for the organization of the wmt biomedical task. the first edition took place in 2016 and addressed scientific abstracts for english/french (both directions), english/spanish (both directions), and en-glish/portuguese (both directions) @xcite . the subsequent shared task included six new language pairs, namely, english into czech, english into german, english into hungarian, english into polish, english into romanian, and english into swedish, in addition to a new type of document, viz., health information texts @xcite . in 2018, we started using med-line® as the source for our scientific abstracts and addressed a new language pair, namely english/chinese (both directions), in addition to some of the languages already considered in the previous year @xcite . in the subsequent year, we introduced the translation of biomedical terminologies (from english into spanish), in addition to the medline abstracts for the five language pairs from the 2018 task @xcite . in 2020, we added three new language pairs, namely english/russian (both directions), english/italian (both directions), and english into basque (en2eu) @xcite . for this year's shared task 1 , we address the same eight language pairs as last year @xcite on the same translations tasks (scientific abstracts and terminologies). the main novel feature this year is a new test set composed of summaries of planned animal experiments to be translated from german into english. the list below summarizes the language pairs addressed this year: finally, we highlight the new aspect that we introduced in the 2021 edition of our shared task, namely, a novel test set for the automatic translation of summaries of animal experiments from german into english (see section 2.4). ## training and test data no additional training data was released for any of the language pairs, with the exception of en2eu, where we provide last year's test set as new training data for abstracts and terminology. as for the tests sets, we released test sets for scientific abstracts, terminologies, and summaries of animal experiments as follows: ## baselines this year we had more choices for the baselines. as before, one option was to use our own models, trained with marian nmt (junczys-dowmunt et al., 2018) on biomedical texts. a new option was to use pre-trained models, not specialized on biomedical texts. we used our own models as baselines for the following language directions: en2de, en2es, en2fr, en2pt, de2en, es2en, fr2en, pt2en. for en2zh, en2ru, en2it, en2eu, zh2en, ru2en, it2en, we used the pre-trained generic marian nmt models available in the huggingface "transformers" library. 9 an interesting question was whether the specialized models were still better than the newest out-of-the-box pretrained models. to this end, for en2de and en2fr we also tested the recent t5large foot_8 model @xcite . for en2fr, it outperformed our own model (trained on biomedical data) by almost 3 bleu points, whereas for en2de the two systems were fairly comparable. the performance of the systems submitted starts from close to baseline for some language directions (e.g. for en2fr, en2es, de2en), whereas for other languages all systems were much better than the baseline (e.g. es2en, pt2en and especially ru2en). 4 teams and systems this year, we received a total of 107 runs from 15 teams from the following countries: china (8), spain (2), france (1), japan (1), pakistan (1), and usa (2). we did not receive any submission for en2pt, even though we did receive submissions from one team for the opposite direction of this language pair, i.e., pt2en. unfortunately, we did not receive any submission for the new test set that we released this year, i.e., for the summaries of planned animal experiments. nevertheless, the test set (including the reference translation) is available for the research community for further experiments. similarly to the wmt 2020 biomedical task edition, we asked participants to fill in a survey with key information regarding the specific material and methods used in their self-identified primary runs that were used for manual evaluation. the survey comprised 14 questions covering the translation methods and corpora used. on average, the time spent by participants to supply information for one language pair was 6 minutes and 35 seconds (median: 3 minutes and 27 seconds). this is consistent with the 2020 survey statistics and suggests that the time commitment for supplying this information is limited, even for teams addressing more than one language pair. all teams used transformer-based neural mt (nmt) and largely relied on existing implementations: 7 teams submitted runs using available libraries while 5 teams submitted runs using their own nmt implementations. teams often used the same setup for a range of language pairs. for in-domain data, teams used the training data distributed as part of the task as well as many of the sources described in @xcite . additional corpus used for chinese have been prepared by the teams but are not always available or described in details. we can also notice that the use or pre-processing of resources supplied by the task organizers can differ between teams as the size reported for seemingly similar data can differ sig- ## automatic evaluation for all the abstracts test sets, we evaluated system outputs using bleu @xcite as provided by the moses tool mteval-v14.pl 11 . we used this metric for the en2eu abstracts, summaries of animal experiments, and medline test sets. in en2zh, a modified version of the tool was used that removed white spaces and text is split in way that each character is a word. the results for the en2eu abstract test sets are given in for the en2eu terminology test sets, we evaluated the translated concepts in terms of two metrics: 11 https://github.com/moses-smt/ mosesdecoder (i) accuracy, by relying on strict matches (case insensitive) between the reference translation and predictions; and (ii) bleu score, as measured by the python nltk module sentencebleu. the results are presented in for the summaries of animal experiments, we only present the results obtained by our baseline system ( finally, for the medline test sets, we performed evaluation based on all the sentences in the test set, including the poorly aligned ones, as well as an evaluation based on only the perfectly aligned ones (see language pair team parallel corpus size (sentence pairs) monolingual corpus size (sentences) de/en huawei_agi medline corpus supplied by wmt biomedical task organizers 2.4 m yes 53 m (en) huawei_tsc medline corpus supplied by wmt biomedical task organizers 3.03m yes 21.43m (en) nrpu-fjwu sources provided by wmt biomedical task organizers (ufal, medline abstracts and emea) 3.71 m no -tmt corpus provided by wmt biomedical task organizers and ufal. 2.5 m yes 2.5 m es/en talp_upc ufal, pubmed, medline, ibecs,uncor-m and opus 6.86 m no -tmt corpus provided by wmt biomedical task organizers and ufal. 1.6 m yes 1.6 m transperfect corpus provided by wmt biomedical task organizers 618 k no fr/en huawei_agi medline corpus supplied by wmt biomedical task organizers 3.6 m yes (en) 53 m lisn bio-medical corpora provided by the task organiser along with taus and cochrane 6 m yes (fr) 0.81 m nrpu-fjwu sources provided by wmt biomedical task organizers e.g. ufal, scielo health, edp, medline titles, medline abstracts and emea. 4.36 m no -tmt corpus provided by wmt biomedical task organizers and ufal. 3.5 m yes 3.5 m it/en huawei_agi medline corpus supplied by wmt biomedical task organizers, taus 374 k yes (en) 55 m mt learner corpus supplied by wmt biomedical task organizers, and in-domain data filtered from an in-house corpus. 364 k yes (en) 1.5 m pt/en mt learner corpus supplied by wmt biomedical task organizers, and in-domain data filtered from an in-house corpus. 1.6 m yes (en) 6.2 m en/ru mt learner corpus supplied by wmt biomedical task organizers, augmented with in-house corpus. 2.2 m yes (en) 2.1 m nvidia corpus supplied by organizers, augmented with automatically filtered news-task corpus. 256k ? ? tmt corpus supplied by organizers, augmented with in-house corpus. 1 m wmt biomedical task and ufal ? transperfect "internal data" (unspecified) 6.1 m no en/zh ecnu_paht in-house corpus (unspecified) 6 m no -huawei_agi in-house data collected from a portion of abstracts of china master's and doctoral dissertations. 847 k no -huawei_tsc in-house corpus (unspecified) 1.35m yes 36.11m (zh), 21.43m (en) transperfect "internal data" (unspecified) with 0.4531 and 0.0.4425 respectively. finally, the nvidia nemo team obtained the best score (0.4139) for en2ru. ## manual evaluation similar to previous years, we manually validated a sample of the abstracts to compare the teams' primary submissions to each other and to the reference translation. for the medline abstracts, we aimed for approximately 100 perfectly aligned sentences and retrieved the corresponding abstracts. the sentences were randomly retrieved, but we aimed to select abstracts with a higher percentage of perfectly aligned sentences. this is the same strategy described in last year's publication @xcite . we only considered those teams which either submitted a publication to the workshop or filled in our survey with information about theirs runs. in some few cases, we could not considered some teams for the manual validation, e.g., mt learner for it2en, because the team filled in the survey when the manual validation was already been carried out. for all teams, we considered the primary run (as indicated by the participants). the only exception was made for the volctrans team, for which we considered the run with the highest bleu score, according to the automatic evaluation. the primary runs that we considered in the manual validation are listed below: abstracts for en2eu (osagaiz + gaceta) were manually validated following the same approach, but only at the sentence level. as there was one submission for this language pair, we only generated a pairwise combination of the participant's run and the reference. the run with the highest bleu score was selected for validation: the translations were evaluated by three annotators using the appraise tool, and the averaged results are presented in the evaluation carried out in this task is compliant with criteria (1-3). however, participants are free to use their choice of training corpus, validation corpus and pre-processing methods. this approach was selected to foster participant creativity and set a lower entry cost to the task. it is a limitation in the comparability of the systems submitted for this task. as a mitigation strategy, we encourage participants to also submit detailed descriptions of system particulars to provide transparency on the material and methods used. a future edition of the task could introduce a "constrained" track where pre-processed training/validation sets would be supplied to be used exclusively (as in the wmt news translation task @xcite ). ## conclusions our sixth edition of the wmt biomedical translation addressed a total of eight language pairs and three types of documents. one more time, we could assess the performance of current mt technology for the translation of biomedical textual resources. further, we could attract the attention of many teams and received submissions for most of our test sets. similar as in the more recent editions of the shared task, participating system could perform better than the reference translation for many of the language pairs. however, this is still a challenge for en2zh. in future editions of this challenge, we aim at releasing more resources, especially additional training data, adding new language pairs, and considering a variety of test sets.