Findings of the WMT 2023 Biomedical Translation Shared Task: Evaluation of C hat GPT 3.5 as a Comparison System (2023)

Abstract: We present an overview of the Biomedical Translation Task that was part of the Eighth Conference on Machine Translation (WMT23). The aim of the task was the automatic translation of biomedical abstracts from the PubMed database. It included twelve language directions, namely, French, Spanish, Portuguese, Italian, German, and Russian, from and into English. We received submissions from 18 systems and for all the test sets that we released. Our comparison system was based on ChatGPT 3.5 and performed very well in comparison to many of the submissions.

## introduction we describe the eighth edition of the biomedical translation task 1 that was part of the eighth conference on machine translation (wmt23). similar to previous years, we released multiple test sets based on biomedical abstracts that we retrieved from the pubmed database. 2 april 26, 2023 and searched for articles that contained abstracts in both english and one of the six languages that we consider. we then randomly selected 100 bilingual abstracts for each of the language pairs. for all language pairs, we split the sentences of the abstracts using scispacy @xcite and aligned them with the geometric mapping and alignment (gma) tool. 4 native speakers of the languages manually checked the alignment quality in the appraise tool @xcite . in this evaluation, we classified the automatically aligned sentences into five categories: we present the results in we released our test sets in two submission systems: (i) our google form as announced on our shared task's web site; (ii) in ocelot, foot_3 both in the general and in the biomedical test sets. model through the openai api. we refer to chat-gpt as a comparison system rather than a baseline, as it does not satisfy the usual criterion for a baseline as being a transparent, well-understood and reproducible model that provides a good (generally) lower bound against which to compare systems. notably, the model is closed-sourced and trained on huge amounts of data, of which the details are not openly known. chatgpt 4 excels at many tasks @xcite , including translation. researchers from tencent identified in a limited early evaluation done before the api was available chatgpt 4 as a good translator @xcite . please note that we abstained from using the stronger chatgpt 4 and used instead the faster but expectedly weaker chatgpt 3.5. more precisely we used the model snapshot "gpt-3.5-turbo-0613", computed on june 13th 2023 but with the training data "up to sept 2021" foot_4 . this reduces the risk of data contamination with respect to the abstracts used in our test sets, which were published in 2023. the chatgpt variants are large and trained on large quantities of data, but are generalist systems. ideally, systems dedicated to translation or specialized in biomedical translation would be able to outperform them, or at least outperform the faster lower-quality version that we proposed here as a point of comparison. otherwise, there are fewer reasons remaining for developing and using an alternative machine translation (mt) system: data privacy, self-hosting, usage in low-resources, nonconnected systems. we used the following prompt to perform the translations and to keep chatgpt from producing any comments beyond the translation text itself: "you are a helpful assistant specialised in biomedical translation. you will be provided with a sentence in {src}, and your task is to translate it into {trg}." where {src} was the source language and {trg} was the target language (e.g. @xmath2. using chatgpt through the api proved to be more challenging than expected and seemed to act as a stress test for the api servers or for the cloudfare content distribution network proxy they use. for example we hit various intentional limitations, such as a rate limit of 90,000 tokens per minute. we then faced multiple other errors: read time out in the object "httpsconnectionpool" with host api.openai.com, http 502 (bad gateway), and "internal error". after writing our api calling code in an idempotent way, we were able to interrupt it whenever it was stuck and restart it whenever we stopped it or it stopped with an error. to this end, the script would skip over the existing translations and proceed with sending for translation, one by one, the rest of the entries not yet translated. the overall experience remained positive, as building the chatgpt 3.5 translations involved 674,470 tokens, resulting in a total api cost of only 1.15 usd. however, we have no information on the co 2 impact of the computation, which should include the impact of inference for translations as well as a fraction of the impact of training the chat-gpt 3.5 model. writing the scripts and executing them took less than three days. the execution itself was fast; as we reported here, at times we exceeded the api limit of 90,000 tokens per minute. ## teams and systems after the release of the test sets, the teams had around two weeks to process the data and submit their translations. we collected submissions from the two systems (our google form and ocelot) belonging to 18 teams (or systems), as listed in this year, the google submission form also included questions on material and methods used by participants. the questions were identical to those used in 2022. the response rate was lower than in previous years (2020-2022) when the questionnaire was operated separately from the submission system and teams were asked to complete the survey after submission. in ocelot submissions, participants were asked to submit a narrative description of their method. none of the teams reported the co 2 impact of their participation in the task. many teams approached the task with transformer-based neural mt (nmt), relying on existing implementations. the use of prompting autoregressive models was also introduced this year. ## automatic evaluation we present bleu scores @xcite for the automatic evaluation in tables 4 and 5 . this includes translations received from both submission systems (google form and ocelot). for both en2de and de2en test sets, the submissions from huaweitsc, zenghuimt, gpt4-5shot, and promt teams obtained higher scores than our comparison system (chatgpt) according to bleu. the bleu scores of the lan-bridgemt submissions (which use gpt3 and gpt4) came very close to those of chatgpt for most language pairs, e.g., en2es, en2it, and were sometimes higher, e.g., fr2en, it2en, and ru2en. most of the online system submissions also got higher bleu scores than chatgpt. however, it is worth bearing in mind the possibility that the online systems had previously seen our test sets in the large data on team id institution biom. task publication airc artificial intelligence research center, japan -(rikters and miwa, 2023) gpt4-5shot microsoft -(hendy et al., 2023) gtcom_peter global tone communication, china -(zong, 2023) huaweitsc huawei translation service center yes (wu et al., 2023) lan-bridgemt lan-bridge communications, china yes (wu and hu, 2023) nllb_greedy (unknown) --nllb_mbr_bleu (unknown) --nrpu_fjwu fatima jinnah women university, pakistan yes (firdous and rauf, 2023) online-a (unknown) --online-b (unknown) --online-g (unknown) --online-m (unknown) --online-w (unknown) --online-y (unknown) --promt promt llc -(molchanov and kovalenko, 2023) upcite-clillf université paris cité, france yes (zhu et al., 2023) ustc_ml_group university of science and technology, china yes -zenghuimt languagex, china -(zeng, 2023) ## manual evaluation we carried out a manual validation of the quality of the translations for some language pairs using the "3-way ranking" task in the appraise tool. it consists of a pairwise comparison with three text spans, for example for en2pt: (i) the source text in english, (ii) translation a in portuguese, and (iii) translation b also in portuguese. the text is either a sentence or the whole abstract, i.e., we carried out the validation for each sentence and then for the complete abstract. the evaluator should choose one of the following four options: (i) @xmath3, i.e., both translations have similar quality; (ii) @xmath4, i.e., translation a is better than translation b; (iii) @xmath5, i.e., translation a is worse than translation b; and (iv) error flag in case one or both of the translations do not refer to the same source text. for the language pairs that we considered, we randomly selected the abstracts until we had at least 100 sentences. we restricted the abstracts to those in which the rate of well aligned (ok) sentences was at least 80%. we considered all pairwise combinations from the following translations: (i) the reference translation, as originally available in pubmed, (ii) translations from chatgpt 3.5, and we present the results in tables 6 and 7 . we compute a significance test (wilcoxon test) when comparing the systems (or reference translation) and we show in bold and with a star (⊛) those cases in which one system (or the reference translation) was better than the other one. none of the teams could outperform the reference translation for all of the language pairs. further, for all language pairs that we checked, the quality of the translations from chatgpt was similar to the reference translation at the sentence level, i.e., there was no significant difference in the results. however, on the abstract level, the chatgpt translations were found to be better than the reference translations for some language pairs, namely, en2ru and fr2en. for some of the languages (e.g. en2de), the rankings from the automatic and manual translations appear consistent. the bleu score from the huaweitsc team was much higher than the one from lan-brigdemt (0.43 versus 0.35), and indeed, the quality of the translations from huaweitsc was better than the ones from lan-brigdemt. there are however some differences in rankings. for example the manual rankings do not correspond exactly to the automatic rankings for ru2en, fr2en and en2it. notably, chatgpt appears to be penalised by bleu and does better in the manual rankings. ## conclusions we presented the finding of the edition of the wmt biomedical translation task. we received submission from 18 systems and compared them to translations from chatgpt 3.5. in the automatic evaluation, some systems were scored higher than bleu according to the comparison system (chatgpt 3.5). in the manual evaluation, none of the systems were systematically better than the reference translation for all of the language pairs that we evaluated. however, in a couple of cases, namely, for fr2en and en2ru, the translations from chatgpt were preferred over the reference translations. we presented a details discussion of the errors that we found during the manual evaluation.