TUD a at WMT 21: Sentence-Level Direct Assessment with Adapters (2021)

Abstract: This paper presents our submissions to the WMT2021 Shared Task on Quality Estimation, Task 1 Sentence-Level Direct Assessment . While top-performing approaches utilize massively multilingual Transformer-based language models which have been pre-trained on all target languages of the task, the resulting insights are limited, as it is unclear how well the approach performs on languages unseen during pre-training; more problematically, these approaches do not provide any solutions for extending the model to new languages or unseen scripts—arguably one of the objectives of this shared task. In this work, we thus focus on utilizing massively multilingual language models which only partly cover the target languages during their pre-training phase. We extend the model to new languages and unseen scripts using recent adapter-based methods and achieve on par performance or even surpass models pre-trained on the respective languages.

## introduction in machine translation (mt), the quality estimation (qe) task attempts to characterize the quality of a translation, without the availability of a (goldlabel) reference translation. the introduction of a qe system would consequently allow for the automatic analysis of machine-translated sentences without costly human reference translation, with numerous applications, such as: the selection of candidate translations, the estimation of human editing effort, or the detection of low-quality or misleading translations @xcite . however, in order to acquire training data, professional human translators are required to score the translation quality of many examples, making labeled data difficult to obtain, especially for low-resource languages. this highlights the importance of crosslingual zero-shot transfer of qe systems, one of the objectives of the wmt21 shared task @xcite , which introduces zero-shot evaluation sets of four new language pairs. previous approaches have predominantly focused on languages for which training data is available, such as the qe task at wmt20. the best results were obtained by fine-tuning massively multilingual transformer-based language models @xcite such as multilingual bert (mbert) @xcite or xlm-r @xcite @xcite @xcite , inter alia), on the target qe tasks. these supervised methods considerably outperform unsupervised methods @xcite @xcite @xcite even in zero-shot settings @xcite . however, analyzing the applicability of fine-tuning multilingual models on the target language pairs that are covered during pre-training considerably limits the generated insights. they are only applicable to the ∼100 languages covered during pre-training, excluding the remaining majority of languages as the "curse-of-multilinguality" @xcite prohibits the over 7000 languages in the world @xcite to be represented within a single model in this work, we thus aim to address these limitations by utilizing multilingual language models that only cover a subset of the target languages. here we focus on mbert which-in contrast to xlm-r-has not seen the languages sinhala, pashto, and khmer, all part of the wmt21 shared task. as the script of sinhala and khmer are not included in the mbert vocabulary, it is impossible for the corresponding tokenizer to correctly tokenize text in those languages. following @xcite we thus propose an adapter-based approach to extend mbert to new languages and new scripts. our contributions are as follows: 1) we analyze adapter-based supervised approaches for qe and demonstrate their competitive performance compared to full model fine-tuning, both in supervised as well as zero-shot settings; 2) we use recent adapter based methods to extend mbert to unseen languages and scripts, achieving considerable performance gains over standard mbert for unseen languages; 3) we demonstrate competitive performance of our adapted mbert approach compared to xlm-r, which has seen the respective languages during pre-training. we release our code and adapters at https://github.com/ aaronsom/wmt21-qe-tudarmstadt/ . ## method we describe our adapter-based approaches for supervised qe and the extension to unseen languages. ## data the sentence-level direct assessment task of wmt21 builds upon the data of wmt20 task 1 @xcite . the wmt20 dataset consists of seven language pairs ranging from the high-resource english-german (en-de) and english-chinese (en-zh), to the medium-resource romanian-english (ro-en), estonian-english (et-en) and russian-english (ru-en), and the lowresource sinhalese-english (si-en) and nepalese-english (ne-en). for each pair, sentences in the source language are sampled from wikipedia (or in the case of russian, from wikipedia and reddit), translated with fairseq @xcite to the target language, and then annotated by at least three professional translators with direct assessment (da) @xcite . the da scores are z-normalized for each annotator and averaged to form the final score. for each of the seven language pairs, the dataset contains 7000 training pairs and 1000 test and dev pairs. the wmt21 dataset extends the wmt20 dataset by providing new test sets-with unpublished labels-consisting of 1000 sentences for each language pair of the wmt20 dataset. in addition, they provide testsets for four new language pairs for zero-shot evaluation, each compris-ing of 1000 sentence pairs with unpublished labels: english-czech (en-cs), english-japanese (en-ja), pashto-english (ps-en), khmer-english (km-en). ## experiments we describe our experimental setup along with the training and implementation details. training & model hyperparameters. we initialize our models with mbert and xlm-r (both large and base-sized). we use a reduction factor r of 8 for our task adapters. language adapters use @xmath0. the additional embeddings for khmer and sinhala contain 10k tokens each and have been fine-tuned together with the respective las on the wikipedia data. we fine-tune our models using adamw @xcite with a linear learning rate schedule without warm-up. we simulate early stopping by storing the checkpoint with the best dev set performance-evaluating every 500 steps. for all models, we use a learning rate of 1e-4 and a batch size of 8. we train each model for 8k steps. hyperparameters have been chosen based on the wmt20 dev set performance. we have chosen the above hyperparameters from the following values ranges: learning rate {5e-5, 1e-4, 2e-4, 5e-4}, batch size {4, 8, 16, 32, 96}, reduction factor r for the task adapters {4, 8, 16}, and training steps {2k, 3k, 5k, 8k, 10k}. ## results & discussion we present the pearson correlation results for our models on the wmt21 test set. the reported values are obtained from the codalab competition. foot_0 ## conclusion in this work, we proposed the use of adapters to fine-tune massively multilingual transformers for the sentence-level qe task. we demonstrated that adapters are able to achieve competitive results with fully fine-tuned models. however, as fully fine-tuned approaches are limited to the languages seen during pre-training, we have employed recent language extension methods to integrate languages unseen by mbert. we extended mbert with language adapters and monolingual embeddings for sinhala, khmer, and pashto. these methods greatly improved the zero-shot performance of the model and largely closed the gap to xlm-r which has been pre-trained on all languages appearing in wmt21. this demonstrates that our approach is applicable, not only to languages seen during pretraining, but also to unseen languages, even with unseen scripts. this suggests that our method is able to extend multilingual models to a wider range of language not covered during pre-training. we suggest that future shared tasks should consider disentangling languages which massively multilingual language models have been pretrained on, from those that are unseen during pretraining, to more closely reflect realistic scenarios, as the majority of languages cannot be represented within a single model @xcite .