Findings of the 2023 Conference on Machine Translation ( WMT 23): LLM s Are Here but Not Quite There Yet (2023)

Abstract: This paper presents the results of the General Machine Translation Task organised as part of the 2023 Conference on Machine Translation (WMT). In the general MT task, participants were asked to build machine translation systems for any of 8 language pairs (corresponding to 14 translation directions), to be evaluated on test sets consisting of up to four different domains. We evaluate system outputs with professional human annotators using a combination of source-based Direct Assessment and scalar quality metric (DA+SQM).

## introduction the eighth conference on machine translation (wmt23) 1 was held at emnlp 2023 and hosted a number of shared tasks on various aspects of machine translation (mt). this conference built on 17 previous editions of wmt as a workshop or a conference @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . following last year's shift from focusing mainly on the news domain, we have continued to explore the capabilities of "general machine translation". in addition to language pairs evaluated last year: @xmath0, english↔chinese, @xmath1, english↔german, english↔japanese, english↔russian, @xmath2, we introduce a new language pair to wmt, namely: english↔hebrew. other than language pairs, there are several differences with respect to last year's task. all language pairs are provided with the sentence boundaries marked except for english↔german, where we decided to experiment with paragraph-level translation. another significant change for this year is the unification of our human evaluation protocol. we no longer rely on reference-based mturk evaluation and move the evaluation towards source-based da+sqm evaluation (introduced last year) with professional annotators. finally, this year's shared task included an increased number of test suites (section 6), allowing the evaluation of mt outputs from different perspectives, including a range of linguistic phenomena, purposely difficult sentences, specialist domains, gendered translations and non-standard ugc translation. all general mt task submissions, sources, references and human judgements are available at github foot_0 . the interactive visualization and comparison of differences between systems can be browsed online on an interactive leaderboard foot_1 using mt-compareval @xcite . the structure of the paper is as follows. we describe the process of collecting, cleaning and translating the test sets in section 2 followed by a summary of the permitted training data for the constrained track section 3. we list all submitted systems in section 4. the human evaluation approach of da+sqm is described in section 5. finally, section 6 describes the test suites and summarises their conclusions. ## test data in this section, we describe the process of collecting data in section 2.1, followed by the explanation of preprocessing steps in section 2.2. producing human references is summarized in section 2.3 and lastly test set analysis is conducted in section 2.4. ## training data similar to the previous years, we provide a selection of parallel and monolingual corpora for model training. the provenance and statistics of the selected parallel datasets are provided in appendix in ## system submissions this year, we received a total of 72 primary submissions from 17 participants. in addition, we collected translations from online mt systems across all language pairs. online system outputs come from 6 public mt services and were anonymized as online-{a,b,g,m,w,y}, which added additional 77 system outputs. the participating systems are listed in finally, we added translations by three contrastive systems. two of them are based on the nllb translation model @xcite modified by @xcite to have a suboptimal performance, using (i) greedy search (nllb_greedy) and (ii) following minimum bayes risk decoding (mbr) optimizing the bleu metric (nllb_mbr_bleu). neither of them is the official (and better performing) nllb model. the third contrastive translation is produced by the large language model gpt4 using 5-shot prompting with fixed random translation examples, using the exact prompt by @xcite together with their predefined few-shot examples. for languages not evaluated in their study, we took examples from the last wmt test sets. appendix e provides details of the submitted systems if the authors provided such details. ## human evaluation human evaluation for all language translation directions is performed with source-based ("bilingual") direct assessment (da, @xcite of individual segments in document context with scalar quality metrics (sqm) guidelines, mostly following the setup established at wmt22 (da+sqm, @xcite . da+sqm asks the annotators to provide a score between 0 and 100 on a sliding scale, but the slider is presented with seven labelled tick marks, as demonstrated in figure 1 . two different annotation platforms and four distinct pools of annotators ( we keep the selection process of documents for annotation mostly the same as in the previous year. the only change made in order to align closer with the mqm-based evaluation run at the metrics shared task @xcite is to present the first 10 segments from a document instead of random 10 consecutive segments. we again collect both segment-level scores and document-level scores, but compute rankings based on segment scores only. ## test suites as can be seen in the general mt task, the improvement of translation quality has made it difficult to discriminate mt output from human translation with the current evaluation methods. nevertheless, there are still cases where mt has difficulties, delivering outputs which despite seeming fluent and being surrounded by other seemingly perfect translations, entail serious flaws. in general evaluation methods, such flaws can get "hidden in the average" or simply get missed altogether. in an effort to shed light to these cases, evaluation via test suites is embedded in the shared task. ## conclusions the general machine translation task at wmt 2023 covered 14 translation pairs, where the only non-english language pair was @xmath5. source based da+sqm was the main human golden truth. the evaluation included 72 primary submissions from 17 participants, 6 online systems and 3 additional contrastive systems including gpt4. it was performed by 155 human (semi-)professional annotators, who contributed more than 175,000 judgments altogether. for most language pairs (apart from @xmath6, mt systems produce outputs that cannot be identified as being worse than the manually produced references translations in a statistically significant way, using our current evaluation methods. it is apparent that this year, the amount of unconstrained submissions are lower thank in past years (27 submissions by 11 participants). additionally, for some language pairs there are only few submissions by participants, and therefore they are dominated by many online systems, of whom we have no technical descriptions. we are therefore considering ways to encourage participation in the future, whereas redefining the constrained setting may be needed. it is the first time that large language models (llms) are included in the shared task as translation systems. although the technology is very apparent in nlp research, we received only one submission using llm methods (lan-bridgemt), whereas one dominant commercial llm (gpt4) was included via our own efforts. gpt4 was in the first significance cluster for all systems translating towards english, but fell in the second significance cluster (rank 3-5) for @xmath7, whereas a similar sign was given by one of the test suites for @xmath8; @xcite . additionally, test suites providers noted that gpt4 outputs are not always faithful to the source sentence (bawden and sagot, 2023) and that they have some issues with speaker gender translation @xcite and specific domains (mukherjee and shrivastava, 2023, e.g. legal;). due to the closed-source nature of commercial tools, it is hard to know the exact reasons for these findings, although they confirm previous observations that gpt models have difficulties with under-represented languages @xcite . we believe that a more transparent comparison including open source llms should be sought for the future.