Findings of the WMT 2020 Shared Task on Machine Translation Robustness (2020)

Abstract: We report the findings of the second edition of the shared task on improving robustness in Machine Translation (MT). The task aims to test current machine translation systems in their ability to handle challenges facing MT models to be deployed in the real world, including domain diversity and non-standard texts common in user generated content, especially in social media. We cover two language pairs – English-German and English-Japanese and provide test sets in zero-shot and few-shot variants. Participating systems are evaluated both automatically and manually, with an additional human evaluation for ”catastrophic errors”. We received 59 submissions by 11 participating teams from a variety of types of institutions.

## introduction in recent years, machine translation (mt) systems have seen great progress, with neural models becoming the de-facto methods and even approaching human quality in news domain @xcite . however, like other deep learning models, neural machine translation (nmt) models are found to be sensitive to synthetic and natural noise in input, distributional shift, and adversarial examples @xcite @xcite @xcite . from an application perspective, mt systems need to deal with non-standard, noisy text of the kind which is ubiquitous on social media and the internet, yet has different distributional signatures from corpora in common benchmark datasets. following the first shared task on machine translation (mt) robustness, we now propose a second edition, which aims at testing mt systems' robustness towards domain diversity. specifically, this year's task aims to evaluate a general mt system's performance in the following two scenarios: we describe the dataset and the task setup in section 3. the shared-task attracted a total of 23 submissions from 11 teams. the teams employed a variety of methods to improve robustness. a specific challenge was the small size of the in-domain noisy parallel dataset. we summarize the participating systems in section 4 and some trends on approaches used by various systems in section 4.1. the contributions were evaluated both automatically and via a human evaluation and the results discussed in section 5. we hope that this task leads to more efforts from the community in building robust mt models. ## related work domain mismatch is a key challenge in machine translation @xcite . most approaches for improving robustness of mt systems to domain shift assume the existence of significant amounts of parallel data in both the source and target domain. in this scenario, a common approach is to first train an mt system on a (generic) source domain and then to fine-tune it on a (specific) target domain @xcite @xcite , to continuously fine-tune on datasets increasingly similar to the target domain @xcite , or to dynamically change the balance of data towards the target domain @xcite . another approach trains a system on multiple domains at the same time, while adding domain-specific tags to the input examples @xcite . both these approaches were employed by participants of the first shared task on mt robustness @xcite . other methods for domain adaptation of mt systems include instance weighting @xcite , incorporating a domain classifier @xcite , and data selection @xcite . some make use of monolingual data available either in the target domain-for example by training the decoder on such data @xcite or by backtranslating it @xcite -or in the source domain, via similar techniques @xcite . @xcite provide a broad survey of domain adaptation for neural mt, which demonstrates that the predominant setup assumes knowledge of the target domain and availability of target domain data at training time. in light of this prior work, the shared task proposed a relatively underexplored scenario, where examples in the target domain are either unavailable or relatively few. other aspects of robustness are robustness to adversarial examples or noisy inputs. the fragility of neural mt models has been previously demonstrated in various settings @xcite @xcite . @xcite proposed a new dataset (mtnt) to test mt models for robustness to the types of noise encountered in the internet. the previous iteration of the shared task focused on robustness of mt systems to such noise @xcite . we refer to that report for more details. ## task to facilitate comparability with the news translation task and also to reduce the participation cost, we suggest the same training data as the wmt20 news task. 1 the focus of the robustness task is to both evaluate models built on this type of data on more challenging test sets, as well as to encourage participants to explore novel training and modeling approaches so that models have more robust performance at test time on multiple domains, including unseen and diversified domains. we offer two language pairs: english-german (@xmath0, with different test sets focusing on one or both these language pairs, or one particular language direction. ## participants and system descriptions we received submissions from 8 teams participating across different tasks, test sets and languages we provided this year. below we briefly present the systems we were able to get a system description paper for: naver labs (nle): they participated in chat and biomedical tasks along with the robustness task. they trained a general big-transformer model using fairseq toolkit @xcite and adapted it towards different tasks using lightweight adapter layers for each task @xcite . they compared results against the more traditional finetuning method @xcite to show that the former provides a viable alternative, while significantly reducing the amount of parameters per task. they also explored using embedding from pre-trained language models in their nmt system of which they tried two mlm variants: i) using nmt encoder's setting, using roberta @xcite . the latter was found more robust to novel domains and noise. the authors found that initializing with first 8 layers instead of the entire model to be optimal. another notable finding included the use of single bidirectional model instead of monodirectional models to give similar performance. for the robustness task specifically they added source side synthetic noise and used bpe drop-out. while this was found to be useful in handling noisy data, no gains towards domain robustness were observed. limsi: limsi participated in biomedical and robustness tasks. for the robustness challenge their main exploration was using adapter layers @xcite applied on 8 domains (parallel data released in the news task). the architecture adds an additional, domain-specific layer on top of every layer of the encoder and the decoder. this allows the test sets from known domains to use adapter layers and for novel domains to use the generic system. they created a noisy domain by adding synthetic noise to source data. the idea is that residual adapter layer learned from such data learns how to deal with noisy domain and is also able to preserve performance on the cleaner domains. however this did not work as well. the residual adapter fine-tuned using the paracrawl corpus gave better performance. uedin: team uedin also mainly trained their system towards news translation task, but added gumbel noise to the output layer of the systems submitted to the robustness task. they followed standard nmt training pipeline and boasted their systems with additional data filtered from the paracrawl corpus. the data was carefully selected using dual cross-entropy (junczys-dowmunt, 2018) and length-normalized cross-entropy. oppo: team oppo also trained their systems for the language pairs released for the news translation task and did not carry any specific exploration towards the task of robustness. their systems followed standard training regime of training transformer models with marian toolkit, with backtranslation to generate synthetic data and ensembles of models. as additional module, they added to their system a reranker trained on six forward and backward models, the scores of which are used as features in training the reranker. prompt: team prompt also participated mainly in the news translation task. their systems were trained using opennmt @xcite toolkit. they applied several stages of data preprocessing including length-based filtering, removing duplications, and using in-house classifier based on hunalign aligner to identify and discard non-parallel sentences. they used two types of synthetic data to improve their systems: i) randomly selecting subset of wikipedia equal to the size of news data and generating parallel corpus through back-translation, ii) creating synthetic data with unknown words using the procedure described in @xcite . systems were trained with tags to differentiate between original data and synthetic data from each other. named entities were handled through a post-processing module with redecoding whenever a named entity was not translated or translated incorrectly. online systems: we also evaluated three top performing online mt systems, which are also commonly used in the wmt news translation task: online-a, online-b, and online-g. while we do not have access to details of the architectures of these models, to the best of our knowledge they are are all neural mt models with one case including a selection between translations from statistical and neural models. ## results in this section we describe the results of both automatic and manual evaluation of general translation quality (section 5.1), as well as an analysis of catastrophic errors (section 5.2). ## conclusions the second edition of this wmt shared task focused on testing mt systems in more challenging conditions than last year, in two ways: (i) by making this in a zero-shot setting, where no training set and no in-domain development set were provided, (ii) by biasing the selection of the test sets to make them even harder to translate, for example, by oversampling segments with toxic content. we hoped to encourage participants in the other wmt translation tasks to submit to this task. indeed, most participating teams submitted standard nmt models trained on other types of data and other wmt tasks. very few teams introduced specific techniques for robustness, such as augmenting training data with synthetic noise. perhaps not entirely surprisingly, strong online systems, which are trained on a large variety of text types and domains, performed well according to both automatic and human evaluation. the only few-shot submission, however, managed to outperform online systems in most test sets, even in those from a different domain from the small training set provided. this is an interesting outcome and shows that few-shot settings are promising. a new protocol was used for human evaluation: for general quality, direct assessment was replaced by likert scores with more detailed guidelines. the ranking of systems according to this human evaluation does not always agree with that given by bleu, which is not surprising. according to human evaluation, systems were ranked together more often. in addition to general quality, we also introduced a flag for catastrophic errors, which is a novel way to evaluate translations. the proportion of sentences containing such errors seems a lot higher than expected. this could be an artefact of the perception of human annotators on what constitutes a catastrophic error. this would explain why even the reference translations are found to contain such errors, albeit on a much smaller scale. in future work we will carry out in depth analysis on the annotation to investigate this high number of catastrophic errors in human and machine translations. etranslation nle oppo uedin limsi promt online-b ref 0 5 10 15 20 25 30 35 40 0 0.02 0.04 0.06 0.08 0.1 tox saf nam neg sen num counts proportion (a) set1: @xmath1.05 0.1 0.15 tox saf nam neg sen num counts proportion (b) set1: @xmath2.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 tox saf nam neg sen num counts proportion (c) set2: @xmath3.02 0.04 0.06 0.08 0.1 tox saf nam neg sen num counts proportion (d) set2: @xmath4.01 0.02 0.03 0.04 0.05 0.06 tox saf nam neg sen num counts proportion (e) set3: @xmath5: distribution of different types of catastrophic errors for all systems excluding oth: absolute count or each error type per system, as well as proportion of sentences in each system that contain that error.