Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications (2021)

Abstract: Sentence-level Quality estimation (QE) of machine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson correlation with human labels. Recent QE models have achieved previously-unseen levels of correlation with human judgments, but they rely on large multilingual contextualized language models that are computationally expensive and make them infeasible for real-world applications. In this work, we evaluate several model compression techniques for QE and find that, despite their popularity in other NLP tasks, they lead to poor performance in this regression setting. We observe that a full model parameterization is required to achieve SoTA results in a regression task. However, we argue that the level of expressiveness of a model in a continuous range is unnecessary given the downstream applications of QE, and show that reframing QE as a classification problem and evaluating QE models using classification metrics would better reflect their actual performance in real-world applications.

## introduction quality estimation (qe) @xcite is the task of predicting the quality of an automatically translated sentence at test time, without the need to rely on reference translations. formally, given a source sentence, s and a translated sentence, t, the goal of qe is to learn a regression model, m such that m(s, t) returns a score that represents the quality of the translated sentence. there are many important applications of quality estimation, for example, translation companies use qe systems to identify mistranslated sentences, which would reduce post-editing costs and efforts, while online platforms use qe systems as filters to hide poorly translated sentences from end-users. additionally, with the proliferation of mined parallel data obtained from web-crawls as a source of nmt training data @xcite , qe has become an important tool in performing quality control on translations from models trained on noisy training data. the performance of a qe system is usually measured by the correlation between predicted qe and human-annotated qe scores. however, the predictions of qe models are primarily used to make binary decisions @xcite : only translations above a certain qe threshold would be given to a human for post-edition in a translation company, or shown to the user in an online platform. therefore, pearson correlation might not be the best metric to evaluate the actual performance of the qe models in real-world use cases. in recent iterations of the qe shared task at the conference on machine translation (wmt) @xcite , the topperforming qe systems have been built on large multilingual contextualized language models that were pre-trained on huge amounts of multilingual text data. further, these qe models are multilingual and work well in zero-shot scenarios @xcite . this characteristic makes them very appealing for real-life scenarios because it removes the need to train one bilingual model for every pair of languages. however, these neural qe models contain millions of parameters and as such their memory and disk footprints are very large. moreover, at inference time they are often more computationally expensive than the upstream neural machine translation (nmt) models, making them unsuitable for deployment in applications with low inference latency requirements or on devices with disk or memory constraints. in this paper we explore applying compression techniques to these large qe models to yield more practical, lower-latency, models while retaining state-of-the-art (sota) performance. our main contributions and findings are: early work on qe built models on manually crafted features extracted from the source and translated sentences, or confidence features directly from machine translation (mt) systems @xcite . in contrast, sota models are usually trained in an end-to-end manner using neural networks @xcite @xcite , without the additional step of feature extraction. the proliferation of many-to-many nmt @xcite has motivated similar multilingual qe models. these multilingual qe models have exploited large pre-trained contextualized multilingual language models to achieve a previously-unseen level of correlation with human judgments in recent iterations of the wmt qe shared task. for example, the top-performing qe model at wmt 2019 @xcite is a neural predictor-estimator model based on multilingual bert @xcite , while the best qe models at wmt 2020 @xcite are regression models built on xlm-r @xcite . @xcite find that these models generalize well across languages and training a single multilingual qe model is more effective than training a bilingual model for every language direction. unfortunately, these models are computationally infeasible for real-world applications. several approaches have been proposed to address the latency issues of these large contextualized language models. most of these work are based on knowledge distillation @xcite @xcite @xcite where large contextualized language models (teacher) are used to supervise the training of smaller student models @xcite , or pruning, which discards redundant model components. some examples are model weights @xcite , tokens @xcite , encoder layers @xcite and attention heads @xcite . existing work has looked at model compression for other multilingual tasks: @xcite obtained 27x speedup on multilingual sequence labeling without any significant performance degradation, while mukherjee and awadallah (2020) obtain 51x speedups on ner tasks while retaining 95% performance. in our experiments with qe we do not observe the same levels of compression effectiveness, suggesting that the qe is a much harder task for model compression. a call to reframe qe as a classification problem was made by @xcite , based on the perspective that classification is more suitable for real-world use cases and binary classes are easier to interpret than the predicted qe scores. our work suggests the same direction, but now from the perspective of modeling, where we empirically find that the level of expressiveness of a regressionbased qe model is unnecessary and evaluating qe models using classification metrics would better reflect their actual performance in real-world applications. ## background and hypothesis current state of the art qe systems @xcite @xcite . are built on xlm-r @xcite , a contextualized language model pre-trained on more than 2 terabytes of filtered commoncrawl data @xcite . as seen in figure 1 , the model concatenates a pair of source and translated sentences with a separator token in between and appends a special cls token to the beginning of the concatenated string. it then converts the pre-processed string into a sequence of embedding vectors using a pre-trained embedding lookup table. the embedding vectors are then encoded using the multi-head self-attention mechanism as described in @xcite . this step is repeated 12 times for the base version of xlm-r and 24 times for the large version of xlm-r. finally, the neural qe model converts the final encoding of the cls token into a qe score via a multilayer perceptron (mlp) layer. in this work, we follow the neural architecture in @xcite , and experiment with both bilingual models (bl), where each model is trained on data from only one language direction, and multilingual models (ml) that are trained on the concatenated data from all language pairs available in the dataset. we choose mean squared error and binary cross-entropy as the objective functions for the regression and binary classification tasks respectively. we use adamw @xcite for parameter tuning and use a common learning rate of 1e -6 for all experiments. ## model compression techniques given the vast amount of work in the field of model compression, we explore three broad techniques and examine whether they could be successfully applied to compressing qe models. ## experimental settings we report results on the mlqe-pe dataset using pearson correlation for regression and f1 for classification. ## results the baseline results on the mlqe dataset foot_0 used at wmt 2020 are shown in ## conclusions this paper presents a thorough study on the efficiency of sota neural qe models and explores whether recent compression techniques can be successfully applied to reduce the size and improve the latency of these models. our experimental results show that recent compression techniques are inadequate for regression as we observe significant performance degradation on the qe task with little improvement in model efficiency. we argue that the level of expressiveness of a qe model in a continuous range is unnecessary since the outputs of the qe model are usually used to make binary decisions. our results show that it is more appropriate to reframe the qe task as a classification problem, and evaluating qe models using classification metrics would better reflect their actual performance in real-world applications. this enables us to achieve sota performance with tiny and efficient models. our experimental results suggest that compressing large neural qe models for the qe regression task remains a challenging problem, especially in the case of multilingual models, where they start showing higher degrees of performance degradation than their bilingual counterparts.