Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference (2023)

Abstract: Machine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference. State-of-the-art QE systems based on pretrained language models have been achieving remarkable correlations with human judgements yet they are computationally heavy and require human annotations, which are slow and expensive to create. To address these limitations, we define the problem of metric estimation (ME) where one predicts the automated metric scores also without the reference. We show that even without access to the reference, our model can estimate automated metrics (ρ = 60% for BLEU, ρ = 51% for other metrics) at the sentence-level. Because automated metrics correlate with human judgements, we can leverage the ME task for pre-training a QE model. For the QE task, we find that pre-training on TER is better (ρ = 23%) than training for scratch (ρ = 20%).

## introduction quality estimation (qe) is often used in machine translation (mt) production pipelines where we need to make decisions based on the quality of an mt output but where the reference is unavailable @xcite . for example, qe is used in translation companies to decide whether to send a specific mt output for post-editing to a human translator or whether to use it directly @xcite @xcite . in this scenario, an accurate qe system has the potential to save expensive translator effort. however, training qe models usually requires human-annotated judgements of the translation quality @xcite @xcite . these human annotations are scarce and costly, especially for low-resource language directions, and may need to be replicated for new mt systems and domains (de @xcite we investigate if automated mt metrics can be used to reduce the cost of learning qe models. automated metrics can be run with no additional costs and can be used to generate large amounts of qe data. if these metrics correlate well with human judgements, this larger data can be used as a partial substitute for human data during training. data augmentation and synthetic qe data via automated metrics has already been explored @xcite @xcite , though never in the pre-training & fine-tuning fashion. our work is guided by a simple intuition. intuitively, human judgements can be thought of as functions depending on the target sequence and (optionally) the source and reference(s): f human ([s], h, @xcite ). the task of qe is to model f human based only on the source s and hypothesis h. because the function arguments of f human @xcite , h, [r]) resemble those of automated metrics for mt: f metric ([s], h, r), we can use the automated metrics to guide learning of the human quality judgements which are hard to obtain and replicate. generating automated metric scores is limited only by the amount of parallel data which is more abundant. because f metric correlates with human judgement f human @xcite , we can start by estimating f metric and only later fine-tune to f human . we refer to estimating f metric (s, h, r) as metric estimation (me) as a parallel task to quality estimation @xcite . we illustrate our idea of pre-training on the automatic metrics and fine-tuning on human assesments in figure 1 . our model uses a bilstm with the source and hypothesis as the input (with several more features like the decoder confidence and hypothesis variance) to output a single number (metric or quality score). experimentally, we find the idea of mitigating data limitation for qe with me pretraining challenging. thus, we structure our investigation around a set of research questions. first, we try to establish that it is possible to robustly predict automated metrics and explore the associated data requirements. then, concerned with the application and deployment of the me model, we also check how transferrable the model is between different mt systems. we break the research down into the following questions. rq1: can automated reference-based mt metrics be reliably predicted without the reference? rq2: what is the effect of data size on predicting automated reference-based mt metrics? rq3: how detrimental is the transfer between different mt systems for metric estimation models? rq4: can metric estimation be used as a pretraining step for quality estimation? we answer the research questions with experiments on the @xmath0. we confirm that bleu is predictable with ρ = 60.4% sentence-level pearson's correlation and other metrics with ρ = 51.3% (rq1, section 4.2). authentic parallel data is needed for me models but this can be alleviated by using more hypothesis from beam search (rq2, section 4.3). it is possible to train the me system on one mt system and then use it on a different mt system with only a slight loss in performance (rq3, section 4.4). we find that pre-training on translation edit rate (ter) @xcite leads to better results than training on the qe data directly, though this approach does not outperform the state-of-the-art in qe (rq4, section 4.5). 2 our metric estimation model notation. given a source sentence s and its translation, h which is an output of a mt system (@xmath1, we build a regressive me model which outputs a numeric score that is close to the output of an automated metric that is further dependent on the reference: f metric (s, h, r). we distinguish two cases based on our level of access to the mt system: blackbox setting (where we assume access only to the mt system output) and glassbox setting (where we have access to entire mt model). in the later case, we may leverage features from the mt model to improve the me capabilities. model. our main model for me/qe, shown in figure 2 , starts with byte-pair-encoding @xcite the source s and hypothesis h. it is followed by bilstm on top of concatenated source and hypothesis with a separator (s[sep]t). the last hidden state (denoted as bilstm -1 ) is extracted and fused together via concatenation with the internal mt system and other features (see the following list). this is then used in a simple feed-forward layer (ffnn) to output a single number: source s, hypothesis h := mt(s) the first me model is glassbox and the second is blackbox. in the first case, we utilize hand-crafted features and also those from the mt system (function features). both of these models are optimized with mean-squared error against a particular metric. that is, we train separate models for each target metric (comet, chrf, bleurt, bleu, meteor, ter) or human judgements. the additional features ψ are: decoder confidence. low probability mt outputs have overall lower quality @xcite @xcite . the decoder confidence is the hypothesis probability as defined by the model i p(t i |@xmath2, s) which is in practice usually computed in the logspace i log p(t i |@xmath3, s). foot_0 hypotheses variance. intuitively, there are many ways to generate a wrong translation but only a few correct ones @xcite . similar to @xcite , we hypothesise that larger variance between the hypotheses correlates negatively with quality. we therefore use the distances between hypotheses as features for our system. specifically, as shown in figure 7 and formalized with the following, we use the mean distance and also the variance between distances as features. we first consider distances from the current hypothesis to be estimated (h1) to all other hypotheses, and then all hypothesis pairs. avg or var({bleu(h 1 , h j )|h @xmath4 baselines. we use multiple baselines for comparison in the me task. apart from the individual features, they are all optimized to minimize the mse loss with a specific target metric. we also use an off-the-shelf qe model used in wmt21 qe task wmt21-comet-qe-mqm (rei automated metrics. for me we use the following mt metrics. bleu @xcite , chrf @xcite , ter @xcite and meteor @xcite work with lexical or character-level units, commonly in word or character n-grams. comet @xcite uses pre-trained encoders to evaluate the hypothesis at a deeper level. bleurt @xcite is another learned metric for text generation which uses pseudo-label. while automated metrics usually yield only sentence-level scores, qe is done for multiple levels: word, phrase and sentence. however, because of the automatic metric restrictions, we also focus on sentence-level qe in this work. ## experiment setup pipeline & data. we start by translating 500k @xmath5. 4 we use a pre-trained wmt19 model by @xcite . for human scores, we use the train data of wmt21 sentence-level quality estimation shared task @xcite which contains 14k human-direct-assessment annotated segments (5k unique). for the human score prediction, we do not have access to the features in the hypothesis space (because the hypotheses were not generated by the mt system to which we have access) and use forced decoding of the pre-trained model to get a confidence estimate. note that since the data comes from a different mt system, there is a distribution mismatch which may negatively influence the results. we address this in section 4.4. we refer to the two datasets as me and qe, respectively and show the distribution of automated metric scores and human judgements in figure 6 . we perform me on both but qe only on the latter because of the human annotations availability. comet-qe lr tf-idf lr multi mbert me all me text 0 10 20 30 40 50 60 correlation (%) bleu bleurt chrf meteor comet ter human figure 4: correlations with metrics and human judgement of baseline and main metric estimation (me) models. each bar is a separate model trained to predict a particular metric or human judgement. individual features are shown in figure 3. evaluation. we evaluate me model performance with pearson's coefficient with the target metric on a dev set of 10k sentences from wmt14 on the segment-level. note that we care about the magnitude (absolute value) of the correlation and not whether it is positive or negative. for example, ter is expected to correlate negatively with human ranking because higher ter means more errors while higher human scores mean higher translation quality. correlations with human judgement are evaluated on 1k wmt21 sentence-level qe data. specifically, we estimate human z-scores which were computed per-annotator. foot_5 ## results this section first studies single-feature baselines (section 4.1) and then the possibility of robust me model (section 4.2) and data size requirements (section 4.3). the model is then checked on a different mt system output to see transferability between systems and architectures (section 4.4). fine-tuning and evaluation on human data (qe) is done in section 4.5. a natural follow-up experiment on using joint prediction to improve the me model is documented in section 4.6. ## complexity & fluency estimation currently, our model was dependent on mostly the source and the hypothesis. if it had access to only the hypothesis, it could still consider its fluency and other factors in estimating the metric. likewise, having access to only the source would correspond to sentence difficulty/complexity estimation. similarly to @xcite , we explored both of these modes and found very high sentence-level correlations. these high correlations, which are close to the full text-only model's performance (ρ = 56.5%) show that our model is not able to utilize the relationship between the source and the hypothesis and that a more elaborate models should be considered. these results are in line with general findings of @xcite . the models' inadequacy is also shown by the imperfect performance when given access to the hypothesis and the reference, just as the metric has: however, the focus solely on the hypothesis or just the source itself has been confirmed for also other qe systems @xcite ) and our model is not an outlier. ## related work this section discusses how our proposed me task fits in the field of qe. confidence estimation. the task of me has a connection to an older task of confidence estimation @xcite , which predates qe @xcite . in confidence estimation, the goal is to predict the probability of the output being correct. @xcite define correctness as a binary class which is based on two thresholded mt metrics: word error rate and nist @xcite . this is in contrast to the me task which is a regression task (predicting e.g. 0.7 instead of good and segment-level). more recent works use the term confidence estimation more freely to mean essentially the qe task with full training data and model access @xcite . because this term is used also in other contexts, such as in calibration @xcite , we define the task metric estimation to avoid ambiguity. feature-based qe models. @xcite pioneered the work of mainstream mt qe. the quest model uses support vector regression on top of features such as source & target lengths, the number of translations in a phrase table or target sentence language model probability. further research has been devoted to devising good features for mt qe models, such as grammatical ones @xcite , ones based on the decoder @xcite or based on the model embeddings spaces @xcite . deeper qe models. quetch @xcite , nuqe @xcite , deepquest @xcite and others @xcite regress directly from the source and hypothesis texts into a score. notably some systems approach sentence-level qe by aggregating or otherwise utilizing previously-estimated wordlevel qe predictions @xcite . pre-training qe models. closest to our work is computing ter between the hypothesis and its post-edited version @xcite . qe is then trained jointly on the artificial and authentic qe data. other models are first trained on artificial (pretraining) and then authentic data @xcite @xcite . the pre-training task does not have to be tied to qe. large pre-trained language models have also been used for qe @xcite @xcite . in contrast, our pre-training aims not only to acquire better sentence representations in general but specifically to acquire better sentence representations for translation quality score estimation. qe models outside of mt the idea to estimate the quality of a prediction given only itself and the input has also been applied to other nlp tasks. ru-ber @xcite uses an rnn-based model to compute a score for a context-response pair in a dialog that is combined with a reference-based score to obtain the final metric. bert-ruber extends this by using pre-trained representations and grade uses the unreferenced scorer in combination with a module that constructs a conversational graph using conceptnet concepts and reasons over it @xcite . recently different works evaluate responses based on specific quality attributes, for example, groundedness @xcite , or combine them into one quality score @xcite ). for open-ended tasks, reference-free metrics, similar to qe, are more desirable because they make fewer assumptions about how the hypothesis should look like. compared to those tasks, the admissible hypothesis space in mt for a given source sentence is more constrained. ## discussion the experiments have shown that the outputs of automated metrics can be predicted even without access to the reference. pre-training only on ter and not any other automated metric outperformed training from scratch, which highlights the importance of exploring multiple metrics instead of just one or two of the most popular ones. pre-training on ter helped because of the large absolute zeroshot correlation. we are, however, unable to provide an explanation for this zero-shot correlation in the first place. our me approach can also be potentially used for improving models, even outside of the mt field, by providing additional signals through selfsupervision. for example, the me model could be used instead of the decoder probabilities for reranking when decoding with beam-search in generative models. this approach would be similar to using qe in decoding by @xcite but would not require a separate scorer model. ## conclusion we proposed the task of metric estimation for machine translation, as a parallel to quality estimation and attempted to solve it with a baseline bilstm model. we show that it is possible to predict the output of a metric without even seeing the reference (ρ = 60.4% for bleu and ρ = 51.3% for other metrics). the main advantage of this task compared to qe is that the data for training me models to predict a particular metric can be generated from any parallel corpus on which the metric can be run. while pre-training on ter outperformed training from scratch, it did not perform better than the commonly used baseline, comet-qe. future work. despite the negative results, features in the hypothesis space should be more explored for tasks beyond me/qe, such as calibration of self-reported confidence in generative models. metric and quality estimation could also be a part of the mt system itself (e.g. as a separate head) which would alleviate the need for an external me/qe model. the me models should also be evaluated for cross-domain performance, similarly to our cross-system evaluation as motivated by needs of production settings. non-perfect correlation with the metrics when presented with the same input shows the imporants of exploring more complex architectures or optimization approaches.