deep Q uest-py: L arge and Distilled Models for Quality Estimation (2021)

Abstract: We introduce deepQuest-py, a framework for training and evaluation of large and light-weight models for Quality Estimation (QE). deepQuest-py provides access to (1) state-of-the-art models based on pre-trained Transformers for sentence-level and word-level QE; (2) light-weight and efficient sentence-level models implemented via knowledge distillation; and (3) a web interface for testing models and visualising their predictions. deepQuest-py is available at https://github.com/sheffieldnlp/deepQuest-py under a CC BY-NC-SA licence.

## introduction quality estimation (qe) for machine translation (mt) aims to predict how good automatic translations are without comparing them to gold-standard references @xcite . this is useful in real-world scenarios (e.g. computer-aided translation or online translation of social media content), where users would benefit from knowing how confident they should be of the generated translations. qe has received increased attention in the mt community, with shared tasks being organised yearly since 2012 as part of wmt, the main conference in mt research @xcite @xcite @xcite @xcite @xcite . given an original-translation sentence pair, qe scores can be computed in different granularities @xcite . at the word-level, each word in the original and/or translation sentence receives a tag indicating whether it was correctly translated or not (e.g. ok or bad). gaps in the translation could also receive labels to indicate when a word is missing. at the sentence-level, a single continuous score is predicted for each original-translation pair. for example, 0-100 for direct assessments (da, @xcite , or 0-1 for human-targeted translation error rate (hter, @xcite . few open-source software is available for implementing qe models. quest @xcite and quest++ @xcite were the first ones, and included methods that relied on extracting linguistically-motivated features to train traditional machine learning models (e.g. support vector machines). with the advent of neuralbased approaches, deepquest @xcite provided a tensorflow-based framework for rnnbased sentence-level and document-level qe models, inspired by the predictor-estimator approach @xcite . openkiwi @xcite implements a common api for experimenting with several feature-based and neural-based qe models. more recently, transquest (ranasinghe et al., 2020b) released state-of-the-art models for sentence-level qe based on pre-trained transformer architectures. as shown in the latest wmt20 qe shared task @xcite , systems are increasingly relying on large pre-trained models to achieve impressive results in the different proposed tasks. however, their considerable size could prevent their application in scenarios where fast inference is required and small disk space is available. to overcome this limitation, @xcite propose to use knowledge distillation @xcite to transfer knowledge from a large topperforming teacher model into a smaller (in terms of memory print, computational power and prediction latency) yet well-performing student model. the authors applied this framework to qe, and effectively trained light-weight qe models with similar performance to sota architectures trained on distilled yet large pre-trained representations. in this paper, we introduce deepquest-py, a new version of deepquest that covers both large and light-weight neural qe models, with a particular emphasis on knowledge distillation. the main fea-tures of deepquest-py are: different from existing open-source toolkits in the area, our aim is to provide access to neural qe models for both researchers (via a command-line interface and python library) and end-users (via a web-based tool). additionally, this is the only tool to provide implementation of knowledge distillation for qe. in the following sections, we detail the main functionalities offered by deepquest-py: implementation of state-of-the-art sentence-level and word-level models (sec. 2); implementation of light-weight sentence-level models through knowledge distillation (sec. 3); and evaluation and visualisation of models' predictions via a web interface (sec. 4). we expect that deepquest-py facilitates the implementation of qe models, allows useful analysis of their capabilities, and promotes their adoption by end-users. ## large state-of-the-art models in the wmt20 qe shared task @xcite , the top performing models were based on fine-tuning pre-trained transformers @xcite , such as bert @xcite or xlm-r @xcite . deepquest-py provides access to this type of approaches by building on the huggingface transformers @xcite foot_0 library. we provide implementations for sentence-level and word-level qe. sentence-level. deepquest-py implements the monotransquest architecture from tran-squest @xcite , the overall winner in task 1 (sentence-level direct assessment) of the wmt20 qe shared task. in this approach, the original sentence and its translation are concatenated using the [sep] token, and passed through xlm-r to obtain a joint representation via the [cls] token. this serves as input to a softmax layer that is used to predict translation quality. in order to boost performance, the authors incorporate two strategies: (1) to use an ensemble of two models: one that fine-tunes xlm-r-base and one that fine-tunes xlm-r-large; and (2) to augment the training data of the qe models with (subsets of) the training data of the nmt models, considering their quality scores as perfect. these extensions are not currently available in deepquest-py, but the api is flexible-enough to incorporate them in the future. word-level. deepquest-py implements the model proposed by beringlab @xcite , the winner of task 2 word-level qe for en-de in the wmt20 qe shared task. similar to the sentence-level models described before, the original sentence and its translation are fed to xlm-r to get contextualised word embeddings. in this approach, both token-level (hidden states) and instance-level ([cls] token) representations are used as input to dedicated linear layers that predict word-level and sentence-level quality estimates, respectively. the model is trained jointly in these two tasks. in order to boost performance, this approach creates artificial data in a similar fashion to @xcite . given a dataset of parallel source-target sentences, an nmt model is trained using 90% of the data. then, the nmt model translates source sentences in the remaining 10% of the data. after that, hter word labels are generated for this 10% of the data, leveraging their manual references as if they were post-edits of the translations generated by the nmt model.deepquest-py includes scripts and examples for executing this training pipeline, provided that the user has access to automatic translations. for obtaining word-level tags, in particular, deepquest-py leverages publicly-available scripts. foot_1 ## light-weight distilled models the most distinctive contribution in deepquest-py is the ability to train light-weight and efficient qe models through knowledge distillation. we implement the approach proposed by @xcite to directly distil sentence-level qe models, where the student architecture can be completely different from that of the teacher. namely, they distill large and powerful qe models based on xlm-r into small bidirectional rnn-based models. ## web tool for analysis and visualisation scores per instance. for sentence-level predictions, each submitted original-translation pair is shown alongside its estimated quality score (figure 3 ). the value for this score and its interpretation depends on the data that the qe model was trained on. for example, models trained on hter scores will output values between 0 and 1, with lower scores indicating better quality. on the other hand, models trained on normalised da scores will output negative and positive values -the higher the better. 4 two additional metrics are included: the proportion of repeated n-grams in the source/target (named source/target n-grams), and the proportion of words in the target sentence that are copies of words in the source (i.e. untranslated words). the user can navigate through the analysed sentences using the form shown (including searching for instances with specific words), or download all the information to a .csv file. ## conclusions we have presented deepquest-py, a new framework for implementation and evaluation of qe models. on top of large state-of-art models based on pretrained transformer architectures, deepquest-py targets the development of light-weight and efficient models around the teacher-student framework for knowledge distillation. in addition, deepquestpy encourages end-user adoption of qe technologies by providing a web application to obtain quality predictions and analyse model performance.