An Exploratory Study on Multilingual Quality Estimation (2020)

Abstract: Predicting the quality of machine translation has traditionally been addressed with language-specific models, under the assumption that the quality label distribution or linguistic features exhibit traits that are not shared across languages. An obvious disadvantage of this approach is the need for labelled data for each given language pair. We challenge this assumption by exploring different approaches to multilingual Quality Estimation (QE), including using scores from translation models. We show that these outperform single-language models, particularly in less balanced quality label distributions and low-resource settings. In the extreme case of zero-shot QE, we show that it is possible to accurately predict quality for any given new language from models trained on other languages. Our findings indicate that state-of-the-art neural QE models based on powerful pre-trained representations generalise well across languages, making them more applicable in real-world settings.

## introduction quality estimation (qe) @xcite is the task of predicting the quality of an automatically generated translation at test time, when no reference translation is available for comparison. instead of reference translations, qe turns to explicit quality indicators that are either provided by the machine translation (mt) system itself (the so-called glass-box features) or extracted from both the source and the target texts (the socalled black-box features) @xcite . in the current qe approaches, black-box features are learned representations extracted by finetuning pre-trained multilingual or cross-lingual sentence encoders such as bert @xcite , * equal contribution. xlm-r @xcite or laser @xcite . these supervised approaches have led to the state-of-the-art (sota) results in this task @xcite , similarly to what has been observed for a myriad of other downstream natural language processing applications that rely on cross-lingual sentence similarity. glass-box features are usually obtained by extracting various types of information from the mt system, e.g. lexical probability or language model probability in the case of statistical mt systems @xcite , or more recently softmax probability and attention weights from neural mt models @xcite . glass-box approach is potentially useful for low resource or zeroshot scenarios as it does not require large amounts of labelled data for training, but it does not perform as well as sota supervised models. qe is therefore generally framed as a supervised machine learning problem, with models trained on data labelled for quality for each language pair. training data publicly available to build qe models is constrained to very few languages, which has made it difficult to assess how well qe models generalise across languages. therefore qe work to date has been addressed as a language-specific task. the recent availability of multilingual qe data in a diverse set of language pairs (see section 4.1) has made it possible to explore the multilingual potential of the qe task and sota models. in this paper, we posit that it is possible and beneficial to extend sota models to frame qe as a languageindependent task. we further explore the role of in-language supervision in comparison to supervision coming from other languages in a multi-task setting. finally, we propose for the first time to model qe as a zero-shot cross-lingual transfer task, enabling new avenues of research in which multilingual models can be trained once and then serve a multitude of languages. the main contributions of this paper are: (i) we propose new multi-task learning approaches for multilingual qe (section 3); (ii) we show that multilingual system outperforms single language ones (section 5.1.1), especially in low-resource and less balanced label distribution settings (section 5.1.3), and -counter-intuitively -that sharing a source or target language with the test case does not prove beneficial (section 5.1.2); and (iii) we study black-box and glass-box qe in a multilingual setting and show that zero-shot qe is possible for both (section 5.1.3 and 5.2). the assumption is that representations learned can, to some extent, account for source complexity, target fluency and source-target adequacy. these are fine-tuned from pre-trained word representations extracted using multilingual or cross-lingual sentence encoders such as bert @xcite , xlm-r @xcite or laser @xcite . @xcite propose the first breakthrough in neural-based qe with the predictor-estimator modular architecture. the predictor model is an encoder-decoder recurrent neural network (rnn) model trained on a huge amount of parallel data for a word prediction task. its output is fed to the estimator, a unidirectional rnn trained on qe data, to produce the quality estimates. @xcite use a similar architecture where the predictor model is replaced by pretrained contextualised word representations such as bert @xcite or xlm-r @xcite . despite achieving strong performances, such models are resource heavy and need to be fine-tuned for each language-pair under consideration. in a very different approach, @xcite propose exploiting information provided by the nmt system itself. by exploring uncertainty quantification methods, they show that the confidence with which the nmt system produces its translation correlates well with its quality. although not performing as well as sota supervised models, their approach has the main advantage to be unsupervised and not rely on labelled data. multilinguality multilinguality allows training a single model to perform a task from and to multiple languages. this principle has been successfully applied to nmt @xcite @xcite . @xcite stretches this approach by translating up to 102 languages from and to english using a transformer model @xcite . they show that multilingual many-to-many models are effective in low resource settings. multilinguality also allows for zero-shot translation @xcite . with a simple encoder-decoder architecture and without explicit bridging between source and target languages, they show that their model is able to build a form of inter-lingual representation between all involved language pairs. shah and @xcite is the only work in qe that attempted to explore models for more than one language. they use multitask learning with annotators or languages as multiple tasks. in a traditional black-box feature-based approach with gaussian processes as learning algorithm, their results suggest that adequately modelling the additional data is as important as the additional data itself. the multilingual models led to marginal improvements over bilingual ones. in addition, the experiments were only conducted with english translation into two closely related languages (french and spanish). ## multilingual qe in this section, we describe the qe models we propose and experiment with. they build upon pretrained representations and represent the sota in qe, as we will show in section 5. pre-trained contextualised representations such as bert @xcite and xlm-r @xcite are deep contextualised language models based on the transformer neural architecture @xcite . these models are pre-trained on a large amount of texts in multiple languages and optimised with self-supervised loss functions. they use shared subword vocabularies that directly support more than a hundred languages without the need for any language-specific pre-processing. we explore qe models built on top of xlm-r, a pre-trained contextualised language model that achieves sota performance on multiple benchmark datasets. baseline qe model (base) given a source sentence s x in language x and a target sentence s y in language y , we model the qe function f by stacking a 2-layer multilayer perceptron (mlp) on the vector representation of the [cls] token from xlm-r: and b 1 ∈ r 4096 . e cls is a function that extracts the vector representation of the [cls] token after encoding the concatenation of s x and s y with xlm-r and relu is the rectified linear unit activation function. we explore two training strategies: the bilingual (bl) strategy trains a qe model for every language pair while the multilingual (ml) strategy trains a single multilingual qe model for all language pairs, where the training data is simply pooled together without any language identifier. we note that this multilingual model here corresponds to a pooled, single-task learning approach. multi-task learning qe model (mtl) multitask learning has shown promising results in different nlp tasks @xcite . here, we want to explore whether having parameter sharing across languages is beneficial, and to what extent having language-specific predictors can boost performance. therefore, we experiment with a simple multi-task approach where we concurrently optimise multiple qe base models that use a language-specific (ls) training strategy. to allow for testing in zero-shot conditions, we also train a language-agnostic (la) component, which receives sampled data from every language. we refer to these two models as mtl-la and mtl-ls. as seen in figure 2 , the mtl-ls submodels and mtl-la submodel share a common xlm-r encoder, while each submodel has its own dedicated language-specific mlp. the intuition of this approach is that it can result in improved learning efficiency and prediction accuracy by exploiting the similarities and differences in the qe tasks for different language directions @xcite . at training time, we iterate through the mtl-ls submodels in a round-robin fashion and alternate between training the mtl-la submodel and training the chosen mtl-ls submodel. at test time, we can evaluate a test set with either the mtl-la submodel or the mtl-ls submodel trained on the same language pair as the test set. ## results in what follows, we pose and discuss various hypotheses on multilinguality for qe. first we focus on our black-box approach from section 3 (section 5.1). second, we examine the behavior of a glassbox approach which does not directly model the source and target texts in multilingual settings (section 5.2). in all cases, we define trainl as the set of language pairs used for training the qe model, and testl as the set of language pairs used at test time. we train different base-bl models for every language pair and a single base-ml model on all language pairs. we also train a single mtl qe model consists of multiple mtl-ls and mtl-la submodels. for each testl, we evaluate it with the mtl-ls submodel trained on the same language pair. we bold the best results across all models. significant improvements over base bl are marked with *. ‡ identifies systems trained on the full 7,000-instance training set with performances reported on the official test set of the wmt'20 qe shared task 1 ( https://competitions.codalab.org/competitions/24447 ), which we assume to come from the same distribution as the dev set. a base-bl model trained on en-zh and tested on en-de performs at average pearson's correlation of 0.37, which is only 0.02 below the best result. we hypothesize that xlm-r might be capturing certain traits in trainl that can generalise well to other testl, i.e. the complexity of source sentences or the fluency of the target sentences @xcite . ## discussion and conclusions from our various experiments, one setting that stood out is that of english-german. we suggest that the difficulty for predicting quality for this language pair was exacerbated by the metric used for evaluation. because of its sample-dependence, pearson correlation can be more sensitive to the output distribution. in contrast, an error-based metric like rmse will be less sensitive to these variations. to illustrate these effects, in figure 6 , we show the hierarchical clustering of language directions obtained by using the metric value from training on one direction and testing on another one as a notion of distance. in subfigure (a), we observe the clusters based on pearson correlation as shown in we explored the use of multilingual contextual representations to build state-of-the-art multilingual qe models. from our experiments, we observed that: 1) multilingual systems are always better than bilingual systems; 2) having multi-task models, which share parts of the model across languages and specialise others, does not necessarily yield better results; and 3) multilingual systems for qe generalise well across languages and are powerful even in zero-shot scenarios. we also contrasted the use of pre-trained representations which are costly to obtain, to the use of glass-box features which can be extracted from the nmt system. we observed that glass-box features are very comparable across languages, and training multilingual systems with them adds little value. finally, we observed that the distribution of the output labels matters for the evaluation of qe.