An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers (2021)

Abstract: Most studies on word-level Quality Estimation (QE) of machine translation focus on language-specific models. The obvious disadvantages of these approaches are the need for labelled data for each language pair and the high cost required to maintain several language-specific models. To overcome these problems, we explore different approaches to multilingual, word-level QE. We show that multilingual QE models perform on par with the current language-specific models. In the cases of zero-shot and few-shot QE, we demonstrate that it is possible to accurately predict word-level quality for any given new language pair from models trained on other language pairs. Our findings suggest that the word-level QE models based on powerful pre-trained transformers that we propose in this paper generalise well across languages, making them more useful in real-world scenarios.

## introduction quality estimation (qe) is the task of assessing the quality of a translation without having access to a reference translation @xcite . translation quality can be estimated at different levels of granularity: word, sentence and document level @xcite . so far the most popular task has been sentence-level qe @xcite , in which qe models provide a score for each pair of source and target sentences. a more challenging task, which is currently receiving a lot of attention from the research community, is word-level quality estimation. this task provides more fine-grained information about the quality of a translation, indicating which words from the source have been incorrectly translated in the target, and whether the words inserted between these words are correct (good vs bad gaps). this information can be useful for post-editors by indicating the parts of a sentence on which they have to focus more. word-level qe is generally framed as a supervised ml problem @xcite trained on data in which the correctness of translation is labelled at @xcite . the training data publicly available to build wordlevel qe models is limited to very few language pairs, which makes it difficult to build qe models for many languages. from an application perspective, even for the languages with resources, it is difficult to maintain separate qe models for each language since the state-of-the-art neural qe models are large in size @xcite . in our paper, we address this problem by developing multilingual word-level qe models which perform competitively in different domains, mt types and language pairs. in addition, for the first time, we propose word-level qe as a zero-shot crosslingual transfer task, enabling new avenues of research in which multilingual models can be trained once and then serve a multitude of languages and domains. the main contributions of this paper are the following: i we introduce a simple architecture to perform word-level quality estimation that predicts the quality of the words in the source sentence, target sentence and the gaps in the target sentence. multilinguality multilinguality allows training a single model to perform a task from and/or to multiple languages. even though this has been applied to many tasks @xcite including @xcite , multilingual approaches have been rarely used in qe @xcite . @xcite explore qe models for more than one language where they use multitask learning with annotators or languages as multiple tasks. they show that multilingual models led to marginal improvements over bilingual ones with a traditional black-box, feature-based approach. in a recent ## architecture our architecture relies on the xlm-r transformer model @xcite to derive the representations of the input sentences. xlm-r has been trained on a large-scale multilingual dataset in 104 languages, totalling 2.5tb, extracted from the commoncrawl datasets. it is trained using only roberta's @xcite masked language modelling (mlm) objective. xml-r was used by the winning systems in the recent wmt 2020 shared task on sentence-level qe @xcite @xcite . this motivated us to use a similar approach for wordlevel qe. our architecture adds a new token to the xlm-r tokeniser @xmath0. we then concatenate the source and the target with a [sep] token and we feed them into xlm-r. a simple linear layer is added on top of word @xmath1. the training configurations and the system specifications are presented in the supplementary material. language pair source mt system competition train size de-en pharmaceutical phrase-based smt wmt 2018 25,963 en-cs it phrase-based smt wmt 2018 40,254 en-de wiki fairseq-based nmt wmt 2020 7,000 en-de it fairseq-based nmt wmt 2019 13,442 en-de it phrase-based smt wmt 2018 26,273 en-ru it online nmt wmt 2019 15,089 en-lv pharmaceutical attention-based nmt wmt 2018 12,936 en-lv pharmaceutical phrase-based smt wmt 2018 11,251 en-zh wiki fairseq-based nmt wmt 2020 7,000 ## results the values displayed diagonally across section i of ## conclusions in this paper, we explored multilingual, word-level qe with transformers. we introduced a new architecture based on transformers to perform wordlevel qe. the implementation of the architecture, which is based on hugging face @xcite , has been integrated into the transquest framework @xcite which won the wmt 2020 qe task @xcite ) on sentencelevel direct assessment @xcite foot_1 . in our experiments, we observed that multilingual qe models deliver excellent results on the language pairs they were trained on. in addition, the multilingual qe models perform well in the majority of the zero-shot scenarios where the multilingual qe model is tested on an unseen language pair. furthermore, multilingual models perform very well with few-shot learning on an unseen language pair when compared to training from scratch for that language pair, proving that multilingual qe models are effective even with a limited number of training instances. while we centered our analysis around the f1-score of the target words, these findings are consistent with the f1-score of the target gaps and the f1-score of the source words too. this suggests that we can train a single multilingual qe model on as many languages as possible and apply it on other language pairs as well. these findings can be beneficial to perform qe in low-resource languages for which the training data is scarce and when maintaining several qe models for different language pairs is arduous.