Findings of the 2022 Conference on Machine Translation ( WMT 22) (2022)

Abstract: This paper presents the results of the General Machine Translation Task organised as part of the Conference on Machine Translation (WMT) 2022. In the general MT task, participants were asked to build machine translation systems for any of 11 language pairs, to be evaluated on test sets consisting of four different domains. We evaluate system outputs with human annotators using two different techniques: reference-based direct assessment and (DA) and a combination of DA and scalar quality metric (DA+SQM).

## introduction the seventh conference on machine translation (wmt22) 1 was held online with emnlp 2022 and hosted a number of shared tasks on various aspects of machine translation. this conference built on 15 previous editions of wmt as workshops and conferences @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . for more than a decade, the machine translation (mt) community has focused on the news domain, which has many desirable features for mt evaluation, such as sufficiently long and grammatically 1 http://www.statmt.org/wmt22/ correct sentences that are easy for both professionals to translate (to produce references) and for human raters to evaluate without specific in-domain knowledge. however, with recent advances in mt and potential overfitting on the news domain (with methods such as fine-tuning on past wmt testsets), we decided to open a fresh research direction of testing the "general machine translation" capabilities. how to test general mt capabilities is a research question in itself. countless phenomena could be evaluated, the most important being: another significant change for this year is the redesign of our human evaluation procedure for @xmath0. we introduce sqm-style da rating, improved sampling of sentences for human judgements, and we opt in for using professional raters. in addition to language pairs evaluated yearly, we introduce several new language pairs that have never been evaluated at wmt or other venues: ukrainian↔english, ukrainian↔czech, livonian↔english, yakut↔russian and @xmath1. lastly, with multiple different shared tasks run at wmt evaluating different phenomena over the same language pairs, we proposed to aggregate test sets and ask participants of different shared tasks to also translate test sets from other shared tasks (for shared language pairs), allowing crosstask evaluation of systems on various phenomena. more details are in section 4.2. general mt task submissions and human judgements are available at https://github.com/ wmt-conference/wmt22-news-systems . the interactive visualization and comparison of differences between systems is at http://wmt.ufal. cz using mt-compareval @xcite . the structure of the findings is as follows. we describe process of collecting, cleaning and translating of test sets in section 2 followed by summary of allowed training data for constrained track section 3. we list all submitted systems in section 4. we use two different techniques for human evaluation. reference-based da is used to evaluate languages into english and described in section 5. da+sqm technique used for non-english and from english translation directions is described in section 6. in section 7, we describe our analysis of @xmath2, translation direction containing professional and student produced references. we conclude the findings in section 8. ## test data in this section, we describe the process of collecting data in section 2.1, followed by the explanation of preprocessing steps in section 2.2. producing human references is summarized in section 2.3 and test set analysis is conducted in section 2.5. lastly, section 2.4 describes specific language pairs that are prepared differently. ## training data similar to previous years, we provide a selection of parallel and monolingual corpora for model training. the provenance and statistics of the selected parallel datasets are provided in appendix in ## system submissions in 2022, we received a total of 107 primary submissions 20 and 82 online systems. the participating institutions are listed in ## human evaluation of translation into english as in previous years, reference-based direct assessment (da, @xcite @xcite was employed as the primary method of evaluation for translation into english. da human evaluation has several important features including accurate quality control of crowd-sourcing and standard methods of significance testing differences in ratings for systems. human assessors are asked to rate a given translation by how adequately it expresses the meaning of the corresponding reference translation or source language input on an analogue scale, which corresponds to an underlying absolute 0-100 rating scale. 23 direct assessment is also employed for evaluation of video captioning systems at trecvid @xcite and multilingual surface realisation @xcite @xcite . for evaluation of translation into-english, we use the monolingual configuration of da, where the human evaluator reads and rates the system output translation and compares its meaning to an english reference translation, which was manually translated by a human translator. as recommended in @xcite , we only employ forwardcreated test data to avoid potential bias. since evaluating segments without their context (i.e. the surrounding document) can cause further bias @xcite , we evaluate sentences in turn taken from a single document and system (described as "sr+dc" in previous wmt reports). 24 similarly to last year, for all language pairs for which document context was available, we include it when evaluating translations. note that the ratings are nevertheless collected on the segment level, motivated by the power analysis described in @xcite , as well as better inter-annotator agreement and lower effort described in @xcite . in terms of the manual evaluation for the translation task for into-english language pairs, a total of 428 turker accounts were involved. foot_22 510,451 translation assessment scores were submitted in total by the crowd, of which 187,922 were provided by workers who passed quality control. 26 system rankings are produced from a large set of human assessments of translations, each of which indicates the absolute quality of the output of a system. similar to last year, all rankings for to-english translation were reached through segment ratings presented one at a time in their original document order (sr+dc). as is usual with da assessments, human assessment scores for translations were first standardized according to each individual human assessor's overall mean and standard deviation score. average standardized scores for individual segments belonging to a given system were then computed, before the final overall da score for a given system is computed as the average of its segment scores (ave z in human performance estimates calculated through the evaluation of human-produced reference translations are denoted by "hu-man" in all tables. translations human-c in @xmath3. clusters are identified by grouping systems together according to which systems significantly outperform all others in lower ranking clusters, according to wilcoxon rank-sum test. all data collected during the human evaluation is available at http://www.statmt.org/wmt22/results. html . appendix b shows the official results for the underlying head-to-head significance test for all pairs of systems. ## human evaluation of translation out of english and without english human evaluation for out-of-english and non-english translation directions foot_25 was performed with source-based ("bilingual") direct assessment of individual segments in context similar to the approach described in @xcite . we use open-source framework appraise for the evaluation @xcite . this year, several changes were made to the annotation procedure, the data sampling, and the interface display. in contrast to the standard da (sliding scale from 0-100) used in 2021, this year annotators performed da+sqm (direct assessment + scalar quality metric). in da+sqm, the annotators still provide a raw score between 0 and 100, but also are presented with seven labeled tick marks, as visible in figure 1 . discrete sqm (0-6) was found to correlate well with mqm (multidimensional quality metrics) annotations by @xcite , while internal preliminary experiments suggested that da+sqm helps to stabilize scores across annotators (as compared to da). annotators performing da+sqm annotations at iwslt 2022 human evaluation campaign (anastasopoulos et al., 2022) also provided positive feedback about the annotation format. in previous years, full documents were sampled for annotation. this year we sampled a maximum of 10 consecutive segments from a document (a document "snippet") for annotation. this provides the potential to annotate segments from a more diverse range of documents while still maintaining a similar number of total annotations. up to 10 source segments preceding and following the snippet being evaluated are displayed as static extra context for the annotator in the interface, as presented in figure 1 . as in past years, annotators provide both segment-level scores and document-level scores (in this case it is more accurate to call them snippet-level scores), however only the segment-level scores were used to compute the official rankings. as the english-livonian data was not document-level, those annotations are run with segment-level-only da+sqm. hits (using the amazon terminology of "human intelligence task" to describe an annotation task) contained quality control segments, as described in section 6.2. rankings are computed as described in section 6.4 based on segment-level scores. ## manual error analysis of @xmath4 in addition to the official human evaluation by assigning da scores, an analysis of errors in @xmath5. the evaluation was carried out bilingually, while looking at the original english segment and all of its translations, both machine and human, all mixed together in a random order. the segments were presented in the natural order in the document, and the entire document (news article or review) was available by scrolling down or up. the analysis was performed on the first 100 documents (80 reviews and 20 news articles), containing 603 segments (416 in reviews and 187 in news). all 14 review topics mentioned in section 2.4 are included, although not uniformly distributed. the annotations are publicly available at https://github.com/wmt-conference/ wmt22-news-systems/humaneval/en-hr/ . the errors were not coupled to any quality criterion (adequacy, fluency, readability) -all problematic words found in the translations were tagged as errors, no matter whether they are related to the source language, or are specific to the target language, or both. there was no distinction of error severity ("major", "minor" or similar). all identified errors (issues) were tagged by their possible causes and/or plausible explanations of their origin, as in @xcite . some of the identified "issue types" are equivalent to the typical error classes that can be found in mqm or similar schemes (such as "mistranslation", "gender", etc.), while some go beyond that, often including several different intertwining types of errors. some of them involve single words, while others might involve a large group of words. the main difference between such tags in comparison to mqm or similar tags is that they are related to (linguistically motivated) causes of errors, also taking into account differences between source and target language as well as the translation process, and not only to the "symptoms" manifested in the mt output. for example, the most frequent issue is related to "rephrasing", and refers to a sequence of words that is not translated properly for some of the following reasons: 1) the translation of the source words follows the structure of the source language although it should be expressed differently in the target language (rephrasing is needed); 2) rephrasing is needed but incorrectly applied; 3) rephrasing is not needed but is applied, and/or 4) the choice of target words is related to source words but seems random, both in lexical as well as grammatical terms. the issue is manifested by several consecutive different but intertwined types of errors such as case, gender, verb form, mistranslation, function word, omission, addition, word order, etc. incorrect translation of multi-word expressions and collocations falls under this type. overall error rates the ranking is similar to the official direct assessment results presented in another potentially interesting difference is the system online-g, which is clearly ranked as second worst by direct assessment, but less clearly as third worst by error annotation. a potential reason is the different nature of errors in different mt systems discussed below. other differences between the two rankings affect only the mid-range systems which have very close scores in both set-ups. it can be seen that errors were detected both in human and in machine translations, although the error rates are notably lower in human translations. overall error rate is lower than 1% for professional translations and lower than 3% for students' translations, while in mt outputs, the overall error rates range from 12 to 22%. in human translations, error rates are similar for both domains. in mt outputs, however, the error rates are notably higher for reviews than for news, which is not surprising given that there are much less training resources for reviews. furthermore, it can be noted that the rankings would be slightly different if only one of the domains were used: ni-utrans would be ranked higher on news while online-g would be ranked higher on reviews and huaweitsc would be ranked lower. nevertheless, those variations in rankings can be observed only for the mid-ranged systems where differences in error rates are small anyway. ## conclusions the general machine translation task at wmt 2022 covered 21 translation pairs, 15 of which had english on the source or target side and 6 were without english. direct assessment (da) was the main golden truth, although the style varied across language pairs. into-english translation was evaluated against human reference translation, preserving the order of sentences in a document but not presenting the whole document at once (sr+dc). out-of-english and non-english pairs offered the context to the annotators and allowed them to re-visit the scores assigned to individual segments (da+sqm), evaluating against the source.