T rans Q uest: Translation Quality Estimation with Cross-lingual Transformers (2020)

Abstract: Recent years have seen big advances in the field of sentence-level quality estimation (QE), largely as a result of using neural-based architectures. However, the majority of these methods work only on the language pair they are trained on and need retraining for new language pairs. This process can prove difficult from a technical point of view and is usually computationally expensive. In this paper we propose a simple QE framework based on cross-lingual transformers, and we use it to implement and evaluate two different neural architectures. Our evaluation shows that the proposed methods achieve state-of-the-art results outperforming current open-source quality estimation frameworks when trained on datasets from WMT. In addition, the framework proves very useful in transfer learning settings, especially when dealing with low-resourced languages, allowing us to obtain very competitive results.

## introduction the goal of quality estimation (qe) is to evaluate the quality of a translation without having access to a reference translation @xcite . high-accuracy qe that can be easily deployed for a number of language pairs is the missing piece in many commercial translation workflows as they have numerous potential uses. they can be employed to select the best translation when several translation engines are available or can inform the end user about the reliability of automatically translated content. in addition, qe systems can be used to decide whether a translation can be published as it is in a given context, or whether it requires human post-editing before publishing or even translation from scratch by a human @xcite . the estimation of translation quality can be done at different levels: document level, sentence level and word/phrase level @xcite . in this research we focus on sentence-level quality estimation. as we discuss in section 2, at present neural-based qe methods constitute the state of the art in quality estimation. however, these approaches are based on complex neural networks and require resourceintensive training. this resource-intensive nature of these deep-learning-based frameworks makes it expensive to have qe systems that work for several languages at the same time. furthermore, these architectures require a large number of annotated instances for training, making the quality estimation task very difficult for low-resource language pairs. in this paper we propose transquest, a framework for sentence-level machine translation quality estimation which solves the aforementioned problems, whilst obtaining competitive results. the motivation behind this research is to propose a simple architecture which can be easily trained with different types of inputs (i.e. different language pairs or language from different domains) and can be used for transfer learning in settings where there is not enough training data. we show that transquest outperforms current open-source quality estimation frameworks and compares favourably to winning solutions submitted to recent shared tasks on 15 different language pairs on different aspects of quality estimation. in fact, a tuned version of transquest was declared the winner for all 8 tasks of the direct assessment sentence level qe shared task organised at wmt 2020 (for more details see @xcite and section 5.1. the main contributions of this paper are the following: the remainder of the paper is structured as follows. we first present a brief overview of related work in order to define the context of our work. in section 3 we present the transquest framework and the methodology employed to train it. the datasets used to train it are presented in section 4, followed by the evaluation and discussion in section 5. the paper finishes with conclusions and ideas for future research directions. ## related work during the past decade there has been tremendous progress in the field of quality estimation, largely as a result of the qe shared tasks organised annually by the workshops on statistical machine translation (wmt), more recently called the conferences on machine translation, since 2012. the annotated datasets these shared tasks released each year have led to the development of many open-source qe systems like quest @xcite , quest++ @xcite , deepquest @xcite , and openkiwi @xcite . before the neural network era, most of the quality estimation systems like quest @xcite and quest++ @xcite were heavily dependent on linguistic processing and feature engineering to train traditional machine-learning algorithms like support vector regression and randomised decision trees @xcite . even though, they provided good results, these traditional approaches are no longer the state of the art. in recent years, neural-based qe systems have consistently topped the leader boards in wmt quality estimation shared tasks @xcite . for example, the best-performing system at the wmt 2017 shared task on qe was postech, which is purely neural and does not rely on feature engineering at all @xcite . postech revolves around an encoder-decoder recurrent neural network (rnn) (referred to as the 'predictor'), stacked with a bidirectional rnn (the 'estimator') that produces quality estimates. in the predictor, an encoderdecoder rnn model predicts words based on their context representations and in the estimator step there is a bidirectional rnn model to produce quality estimates for words, phrases and sentences based on representations from the predictor. to be effective, postech requires extensive predictor pre-training, which means it depends on large parallel data and is computationally intensive @xcite . the postech architecture was later re-implemented in deepquest @xcite . openkiwi @xcite is another open-source qe framework developed by unbabel. it implements four different neural network architectures quetch @xcite , nuqe @xcite , predictor-estimator @xcite and a stacked model of those architectures. both the quetch and nuqe architectures have simple neural network models that do not rely on additional parallel data, but do not perform that well. the predictor-estimator model is similar to the postech architecture and relies on additional parallel data. in openkiwi, the best performance for sentence-level quality estimation was given by the stacked model that used the predictor-estimator model, meaning that the best model requires extensive predictor pre-training and relies on large parallel data and computational resources. in order to remove the dependency on large parallel data, which also entails the need for powerful computational resources, we propose to use crosslingual embeddings that are already fine-tuned to reflect properties between languages. we assume that by using them we will ease the burden of having complex neural network architectures. over the last few years there has been significant work done in the area of crosslingual embeddings @xcite . since the introduction of bert @xcite , transformer models have been used successfully for various nlp tasks such as named entity recognition @xcite , sentence classification @xcite , and question answering @xcite , in many cases improving the state of the art. most of the tasks were focused on english due to the fact that most of the pre-trained transformer models were trained on english data. although there are several multilingual models like multilingual bert (mbert) @xcite and multilingual distilbert (mdistilbert) @xcite , researchers expressed some reservations about their ability to represent all the languages @xcite . in addition, although mbert and mdistilbert showed some crosslingual characteristics, they do not perform well on crosslingual benchmarks @xcite . xlm-roberta (xml-r) was released in november 2019 @xcite as an update to the xlm-100 model @xcite . xlm-r takes a step back from xlm, eschewing xlm's translation language modeling (tlm) objective since it requires a dataset of parallel sentences, which can be difficult to acquire. instead, xlm-r trains roberta @xcite on a huge, multilingual dataset at an enormous scale: unlabelled text in 104 languages is extracted from commoncrawl datasets, totalling 2.5tb of text. it is trained using only roberta's @xcite masked language modelling (mlm) objective. surprisingly, this strategy provided better results in crosslingual tasks. xlm-r outperforms mbert on a variety of crosslingual benchmarks such as crosslingual natural language inference and crosslingual question answering @xcite . both architectures proposed in transquest have been successfully applied in the monolingual semantic textual similarity tasks @xcite . when applied in monolingual experiments, both of them use monolingual transformer models like bert @xcite , roberta @xcite as the input. this inspired us to change the input in such a way that it can represent both the source and target sentences for which the quality of translation needs to be estimated, with the hope that the same architectures would also provide good results in the qe task. our initial experiments showed that crosslingual embeddings like xlm-r provide better results than multilingual embeddings like mbert. therefore, in this research we explore the performance of crosslingual embeddings with simple neural network architectures for the sentence-level quality estimation task. to the best of our knowledge, state-of-the-art crosslingual contextual embeddings such as xlm-r have not been used in quality estimation before. ## methodology this section presents the methodology used to develop our quality estimation methods. we first describe the neural network architectures we proposed, followed by the method used to train these architectures. ## dataset we used the architectures described above to predict two standard measures that express the quality of a translation: human-mediated translation edit rate (hter) and direct assessment (da). all the datasets that we used are publicly available and were released in wmt quality estimation tasks in recent years @xcite @xcite . this was done to ensure replicability of our experiments and to allow us to compare our results with the state of the art. in the reminder of the section we provide more details about the datasets used. ## evaluation and discussion this section presents the evaluation results of our architectures on the datasets described in the previous section in a variety of settings. we first evaluate them in a single language pair setting (section 5.1), which is essentially the setting employed in the wmt shared tasks. we then evaluate in a setting where we combine datasets in several languages for training (section 5.2). we conclude the section with an evaluation of a transfer learning setting (section 5.3). in order to better understand the performance of our approach, we compare our results with the baselines reported by the wmt2018-2020 organisers. two baselines were used: openkiwi @xcite and quest++ @xcite . row iv of tables 2 and 3 present the results for these baselines. for some language pairs in the 2018 wmt quality estimation shared task the organisers did not report the scores obtained using openkiwi. these cases are marked with nr in the evaluation metric used was the pearson correlation (r) between the predictions and the gold standard from the test set, which is the most commonly used evaluation metric in recent wmt quality estimation shared tasks @xcite @xcite . ## conclusions in this paper we introduced transquest, a new open source framework for quality estimation based on cross-lingual transformers. transquest is implemented in pytorch and supports training of sentencelevel quality estimation systems on new data. it outperforms other open-source tools on both aspects of sentence-level quality estimation and yields new state-of-the-art quality estimation results. as far as we know, it is the first time that an open-source qe framework has been tested on both aspects of quality estimation. furthermore, it is the first time that a qe system explores multi-language pair models and transfer learning on low-resource language pairs. unlike many other open-source neural qe frameworks, transquest does not use parallel data and hence does not require similar computational resources. we propose two architectures: mtransquest and stransquest, neither of which have been previously explored in qe tasks. the two architectures have a trade-off between accuracy and efficiency. on an nvidia tesla k80 gpu, mtransquest takes 4,480s on average to train on 7,000 instances, while stransquest takes only 3,900s on average for the same experiment. on the same gpu, mtransquest takes 35s on average to perform inference on 1,000 instances which takes stransquest only 16s to do so. therefore we recommend using mtransquest where accuracy is valued over efficiency, and stransquest where efficiency is prioritised above accuracy. since there is a growing interest 3 in the nlp community for energy efficient machine learning models, we decided to support both architectures in the transquest framework. in the future, we plan to expand transquest with more neural network architectures and more models for different levels of quality estimation such as word-level and document-level. in the sentence-level, we plan to perform transfer learning on language pairs that do not include english at all. we also hope to conduct unsupervised experiments on low-resource language pairs.