Findings of the WMT 2022 Biomedical Translation Shared Task: Monolingual Clinical Case Reports (2022)

Abstract: In the seventh edition of the WMT Biomedical Task, we addressed a total of seven languagepairs, namely English/German, English/French, English/Spanish, English/Portuguese, English/Chinese, English/Russian, English/Italian. This year’s test sets covered three types of biomedical text genre. In addition to scientific abstracts and terminology items used in previous editions, we released test sets of clinical cases. The evaluation of clinical cases translations were given special attention by involving clinicians in the preparation of reference translations and manual evaluation. For the main MEDLINE test sets, we received a total of 609 submissions from 37 teams. For the ClinSpEn sub-task, we had the participation of five teams.

## introduction this is the seventh edition of the biomedical translation task offered under the umbrella of the conference on machine translation (wmt22). 1 this shared task builds on the six previous editions of the biomedical translation task @xcite @xcite @xcite . similar to previous years, we addressed seven language pairs, in both directions, namely: german/english (de2en and en2de), spanish/english (es2en and en2es), french/english (fr2en and en2fr), italian/english (it2en and en2it), portuguese/english (pt2en and en2pt), russian/english (ru2en and en2ru), and chinese/english (zh2en and en2zh). in the biomedical translation task this year, participants were asked to translate shared test sets (described in section 2) comprising documents belonging to three different text genres: scientific abstracts, clinical cases and terminology items. in total, seven language pairs (14 translation directions) were included this year, with both low-resource and high-resource pairs. for each language direction, we provide baseline systems relying on pre-trained neural translation models (described in section 3). in order to help gain insight into the system performance, we collected information on the specific material and methods used in the systems from the participants (section 4). system outputs for each task were evaluated both automatically and manually (as described in section 5 and 6, respectively). one particular growth direction that we explored this year was the inclusion of full clinical case descriptions. a small set of five clinical cases in english were included in the medline test sets from english and a larger clinical corpus corpus was also included in the clinspen track. we also involved clinicians in the preparation of gold standard translations and manual evaluation of clinical cases for en2fr and en2es. two types of submissions were received for the medline test sets: those submitted using (i) our submission system (hereafter called biowmt), as in previous years, and (ii) the ocelot submission system, foot_1 which was also used in the wmt general task. in addition, an independent subtask was held as part of the shared task: clinspen. 3 clinspen focuses on the automatic translation of clinical content in both english and spanish. three subtracks are proposed based on different possible use cases: clinical case reports, clinical terminology obtained from literature and electronic health records (ehr) and ontology concepts. unlike the rest of the tasks, clinspen's evaluation was done through codalab 4 and new submissions can still be made. ## test sets in this section we describe the various test sets that we released for this year's edition of the wmt biomedical task. ## baselines the baselines for en2de, en2fr, en2es, en2pt, de2en, fr2en, es2en, and pt2en were computed using models we trained ourselves in the previous years using marian nmt @xcite . the baselines for en2zh, en2it, en2ru, zh2en, it2en, zh2en were computed using pre-trained marian models distributed as huggingface "transformers" library models, foot_6 without trying to increase their performance on the biomedical texts through further fine-tuning. the computation was performed on a single nvidia a5000 gpu card. the baselines are strongly outperformed by the participants of the biomedical task, with the exception of en2it where all reach similar and very high levels, in excess of 47 bleu. especially our zh2en baseline needs improvement. ## teams and systems in this section we describe the teams and the number of submissions that we received from our two submission systems. when considering both the medline and the clinspen sub-task, we had a total of 40 participating teams. we describe the submissions for each of them below. ## automatic evaluation in this section we present the automatic evaluation that we performed for the medline and the clinspen test sets. ## manual evaluation for the medline test sets, we performed a manual evaluation for some selected runs from some of the teams. in this section we describe how the teams and runs were selected, the results of the manual evaluation, and our observations on the quality of the translations. ## conclusions we presented an overview of this year's edition of the wmt biomedical task. we released test sets for seven language pairs, and addressed a variety of textual sources, such as scientific abstracts, clinical case reports, and terminologies. we had a record number of participating teams and of submissions. all submissions were automatically evaluated in terms of bleu scores, with respect to reference translations, whenever available. we also manually evaluated a selection of the submissions, and similar to previous years, the translations from some teams achieved a similar quality to the reference translations.