Identifying Weaknesses in Machine Translation Metrics Through Minimum B ayes Risk Decoding: A Case Study for COMET (2022)

Abstract: Neural metrics have achieved impressive correlation with human judgements in the evaluation of machine translation systems, but before we can safely optimise towards such metrics, we should be aware of (and ideally eliminate) biases toward bad translations that receive high scores. Our experiments show that sample-based Minimum Bayes Risk decoding can be used to explore and quantify such weaknesses. When applying this strategy to COMET for en-de and de-en, we find that COMET models are not sensitive enough to discrepancies in numbers and named entities. We further show that these biases are hard to fully remove by simply training on additional synthetic data and release our code and data for facilitating further experiments.

## introduction recently, neural machine translation evaluation metrics have reached better correlation scores with human evaluators than surface-level metrics like bleu @xcite . in particular, comet @xcite has shown significant potential as a leading evaluation metric both in shared tasks @xcite and other studies on machine translation evaluation metrics @xcite . the main benefits of such neural metrics are that they do not rely on surface-level similarity to a reference translation and that some of them operate in a multilingual representation space. this also allows for comparing translations to the source sentence. a recent evaluation as part of the wmt 2021 metrics shared task @xcite suggests that neural metrics are also less susceptible to many weaknesses of earlier non-neural metrics, e.g. an antonym in the translation hurting the bleu score exactly the same amount as a synonym. however, it is still unclear whether or not these metrics also introduce new biases that are harder to detect since they are essentially "black box" metrics that do not explain why a certain score is attributed to a translation. failing to identify these biases in neural metrics could lead the community to optimise towards metric "blind spots", either directly through reward-based training methods such as minimum risk training @xcite , or more slowly by basing modelling choices on metric scores. it is therefore worthwhile to find new means to uncover weaknesses of neural machine translation metrics. in this paper, we show that sampling-based minimum bayes risk (mbr) decoding -where a pool of samples are compared against each other using a machine translation evaluation metric as a utility function -can render blind spots of these metrics more observable. when applying comet as the utility function, we find many examples where a translation hypothesis is chosen that contains different numbers or named entities than the source and reference (see examples in our contributions are the following: ## related works how to best evaluate machine translation models has been a long-standing question in the research src schon drei jahre nach der gründung verließ green die band 1970. ## minimum bayes risk decoding traditionally, maximum a posteriori (map) decoding is used in the context of neural machine translation. the goal is to find the translation hypothesis h i among all possible hypotheses h that is most probable under the translation model given the source sentence x and the model parameters θ: in practice, it is not feasible to consider every possible hypothesis. beam search offers a popular and effective approximation. in contrast, mbr decoding aims to find a translation that minimises the expected cost (risk) of choosing a candidate translation h i , assuming that we have some loss function l to compare the candidate to a true translation h j , and access to the true probability distribution p : since we do not have access to the true probability distribution p , and cannot exhaustively sum over all possible translations h, we have to make several approximations. first, we select a subset of all possible hypotheses h as candidate translations c to make the computation tractable. eikema and aziz (2020) suggest drawing ancestral samples from the translation model as a set of unbiased candidates, and we follow this sampling-based mbr approach. ancestral samples s are created by sampling the next token w from the translation model according to the probability distribution over the vocabulary v at each time step t: the probability distribution is conditioned on the source sentence x and the previously produced output tokens s t-1 . for each ancestral sample s, this sampling continues until the end-of-sentence symbol is sampled as the next token w. second, we need to create an additional set of "support hypotheses" s that serve as an approximation to the unknown true translation. the set of candidates c and the set of support hypotheses s can be created separately but in this work, we follow @xcite and let our translation model produce a set of 100 ancestral samples that are used both as candidates and support (@xmath0. third, we need to define a loss function l. in practice, we often substitute the loss function for a similarity function where higher values are better. such a "utility function" u is then used to search for the translation h i that maximises the expected utility or -to paraphrase -is most similar to all hypotheses in the support set s: any automatic machine translation evaluation metric can be used as the utility function u. eikema and aziz (2021) find that beer @xcite works best among a range of nonneural metrics. more recently, @xcite compare several metrics as utility functions in a human evaluation of mbr-decoded outputs where the neural metric bleurt @xcite clearly outperforms non-neural metrics. in this paper, we explore the use of another neural evaluation metric as the utility function, namely comet. since the reference-based comet model takes the source, a translation hypothesis and a reference (approximated in mbr decoding with another hypothesis) as input, our formulation of mbr decoding now takes into account the source sentence x: for an efficiency-related discussion of our implementation, please refer to section 4.3. ## experiment setup 4.1 translation model to be able to generate samples, we train two transformer base machine translation models @xcite using the nematus foot_1 @xcite framework, one from @xmath1. we follow @xcite and use all available parallel data from the wmt 2018 news shared task @xcite except for paracrawl as training data. this amounts to 5.9 million sentence pairs. after deduplication, we have approximately 5.6 million training examples. both models are trained for 250k updates and we choose the best checkpoint based on the bleu score as evaluated on newstest2017 using sacrebleu @xcite . we compute a joint subword vocabulary of size 32k with byte pair encoding @xcite using the sen-tencepiece implementation @xcite . during training and decoding, the maximum sequence length is set to 200 tokens. our models are built with 6 encoder layers, 6 decoder layers, 8 attention heads with an embedding and hidden state dimension of 512 and a feedforward network dimension of 2048. for regularisation, we use a dropout rate of 0.1 for bpedropout @xcite during training, for the embeddings, for the residual connections, in the feed-forward sub-layers and for the attention weights. we train with tied encoder and decoder input embeddings as well as tied decoder input and output embeddings (press and wolf, 2017) and apply exponential smoothing of model parameters (decay 10 -4 ) @xcite . following previous work on mbr decoding @xcite , we train without label smoothing. for optimisation, we use adam (kingma and ba, 2015) with standard hyperparameters and a learning rate of 10 -4 . we follow the transformer learning schedule described in @xcite with a linear warm-up over 4,000 steps. our token batch size is set to 16,348 and we train on 4 nvidia tesla v100 gpus. ## exploration of mbr-decoded outputs we employ sampling-based mbr decoding as a strategy to identify weaknesses in evaluation metrics that are used as utility functions. we believe that -in addition to general errors -we may also find other errors that can stem from two sources: first, since samples are often of lower quality than hypotheses produced with beam search, neural metrics may behave unexpectedly when faced with errors that occur less frequently in beam search based machine translation outputs on which they were trained. second, in mbr decoding, we compare a candidate translation hypothesis to a pseudoreference (another hypothesis) instead of an actual reference. this is also something neural metrics were neither trained on nor designed to do. we are most interested in general errors and errors of the first type since the second type is only relevant for mbr decoding itself. therefore, we conduct additional experiments in section 6 to distinguish between these two sources for the errors we identify below. note that errors of the second type may become more important to investigate as mbr decoding becomes more prevalent or if we evaluate against multiple translation hypotheses instead of references @xcite . in our experiments, we first manually compare mbr-decoded outputs that were chosen with two different evaluation metrics as the utility function: chrf++ and comet. for comet, we notice several cases where the chosen hypothesis contains numbers and named entities that do not match with the source and the reference, even though the majority of samples in the support set contain the correct numbers and named entities. two examples are shown in to test if these findings apply at scale, we run an automatic evaluation. for numbers, we use regular expressions to identify numbers in the mbrdecoded outputs. we measure the overlap between numbers in the source and the translation with the f1-score. we decide to compare to the source to be able to compute the overlaps for the reference and the alternative human translation as well. the results can be seen in the left part of consequently, these results indicate that mbr decoding with the comet metrics chooses more erroneous translations with respect to these criteria than with the two non-neural metrics or compared to beam search decoding. interestingly, the wmt21-comet-mqm model performs considerably worse than the wmt20-comet-da model in this analysis. oracle experiments where we choose the sample closest to the two references according to different metrics (see appendix b) show smaller f1-score differences between both comet models and the non-neural metrics but they still perform worse, particularly compared to chrf++. it is worth noting that the beam search output has the highest f1-score of all tested decoding strategies. this suggests that mistranslations of numbers and named entities do not occur as frequently in beam search outputs and comet's insensitivity to numbers and named entities could therefore be less harmful when evaluating beam search outputs. however, @xcite recently showed that state-of-the-art research models and commercial nmt systems still struggle with numerical translations even when decoding with beam search. such mistranslations may also occur more frequently in out-of-domain and low-resource settings and therefore, we argue that this insensitivity of comet is not only harmful for sampling-based mbr decoding but also when evaluating beam search output. this automatic evaluation has strengthened the findings in our manual exploration that wrong number and named entity translations are recurring problems. to better quantify how sensitive comet models are toward these error types, we propose to perform an mbr-based sensitivity analysis in the next section. ## mbr-based sensitivity analysis our findings in the previous section stand in contrast to the corrupted reference analysis performed as part of the wmt 2021 metrics shared task @xcite where comet mostly preferred the correct alternative human translation to one with swapped numbers when comparing to the reference. in reality, we will seldom have a hypothesis pool with a perfect translation and variants of it that only differ in one aspect. ideally, evaluation metrics should be able to order translation hypotheses with many different error types according to their severity. therefore, it makes sense to compare how much metrics punish different error types. since our previous analysis showed that many samples with number and named entity mismatches are chosen in mbr decoding, this indicates that comet is not as sensitive to these error types as to other errors. to further support this finding, we propose to look more closely at how comet behaves with different error types. as described in section 3, in mbr decoding, every candidate translation is assigned a score that represents the average similarity to the support hypotheses. consequently, if the support is kept constant and a targeted change is made to a candidate translation, the difference in this mbr score indicates how sensitive the utility function was towards this change. we term this an "mbr-based sensitivity analysis". to measure comet's sensitivity towards changes in numbers and named entities, we create a candidate pool that consists of the reference translation and several changed variants. note that the support still contains the same 100 samples that were used to find the mbr-decoded outputs described in section 5. in particular, we make the following targeted changes to the reference to measure the sensitivity towards each change: as reference points, we also apply the same types of changes to random nouns in the reference: additionally, our candidate pool contains the following hypotheses to be used as controls: we use the same tools to identify numbers and named entities as in section 5 to create these perturbations of the reference. for each newly created candidate, we compute the difference to the mbr score of the reference. we then average those differences across sentences for each perturbation type. the results for the sensitivity analysis with the wmt20-comet-da model can be seen in the left part of following our discussion of error sources at the beginning of section 5, it is a valid concern that if we were to compare the candidates to high-quality support translations rather than samples, comet may be more sensitive toward number and named entity differences as there would be fewer other discrepancies between the candidates and the support. to test if this is the case, we repeat the sensitivity analysis but now use the two alternative references as the support instead of the 100 samples that were used before. the candidates are formed by applying the same perturbations as before to the 1-best beam search output instead of the reference. this mimics an oracle setup. the results for this experiment are shown in the middle of the differences in the mbr score of the unperturbed beam search output are generally larger in this setup, which indicates that comet is indeed more sensitive to errors when used as intended, i.e. with high-quality translations and correct references. however, we can still see that the perturbations made to random nouns result in much larger differences than perturbations made to numbers or named entities. this indicates that the problem cannot be attributed to the mbr decoding setting and low-quality pseudo-references alone. ## comet retraining one possible explanation for the low sensitivity of comet to perturbations of numbers and named entities is that these errors are too rare in the wmt outputs used to train comet. we decide to retrain comet on the original training data plus added synthetic data on which we perform the same perturbations as described in section 6. the idea is that the newly trained model is more sensitive to- figure 1 : difference in sensitivity to the same error type applied to a random noun for the de-en test set with samples as support. comparing the original wmt20-comet-da to three retrained models, with different amounts subtracted from the original score for synthetic examples (-0.2, -0.5 and -0.8). to retrain the wmt20-comet-da model, we use the data from the wmt metrics shared tasks collected in the years 2017 to 2019 @xcite @xcite as training data. for every @xmath2, we randomly apply one of the perturbations described in section 6 (except for the perturbations of random nouns and whole named entities). to encourage comet to punish such synthetically inserted mismatches, we modify the scores of the original examples by subtracting a penalty from the z-score of the direct assessment (da) score. we retrain three different models with penalties of -0.2, -0.5 and -0.8 respectively. within every experiment, the penalty is the same for all error classes. the resulting ∼61k synthetic training examples are then added to the ∼640k original examples which means that roughly 10% of the data are synthetic. 11 we follow the hyperparameter suggestions in @xcite for retraining comet but we do not perform model averaging. the models are trained for two epochs and the hyperparameters are listed in appendix a. we ensure that the retrained models still perform as well as the original model on the wmt 2020 metrics shared task @xcite . the average difference in systemlevel pearson correlation to the original comet model lies within 0.006 for all three penalties. the full results can be found in appendix f. 11 we also trained models with larger amounts of synthetic data but did not see an improvement (see appendix e). the effects of retraining with different penalties can be seen in figure 1 (tables in appendix d). subtracting -0.2 from the original scores for synthetic examples can slightly reduce the difference between the mbr scores for numbers / named entities and random nouns with the same error types. retraining with -0.5 subtracted from the original score improves this further but still cannot close this gap completely. with a penalty of -0.8, we now see a larger sensitivity to numbers and named entities than to random nouns for several error types. however, the difference to random nouns is still rather high for substituting a digit in numbers. when repeating the automatic analysis from section 5 with the penalty -0.8 model, we see that retraining does improve the f1-scores (see last row in from this experiment, we conclude that removing such blind spots from comet -once identified -might need more effort than simply training on additional synthetic data. we hypothesise is that the xlm-r component learns very similar representations for numbers and rare words like named entities during pretraining which could be hard to reverse with finetuning only. @xcite show that pretrained language models are surprisingly bad at guessing the correct number from context (e.g. "a bird usually has [mask] legs.") which supports this hypothesis. several other works also find that task-specific models often struggle with numbers and named entities such as in summari-sation @xcite or question answering @xcite . we leave a more extensive analysis of biases in the human evaluation training data (e.g. unpunished number mismatches) and further experiments on weaknesstargeted training for future work. ## conclusion identifying weaknesses of neural machine translation evaluation metrics becomes more important as these essentially "black box" evaluation tools become more popular and are optimised towards during model development. we show that mbr decoding can be used to explore biases of such metrics. through a case study, we show that comet is relatively insensitive to mistranslated numbers and named entities. this can be seen both in the mbr-decoded output which contains a higher number of these errors compared to beam search (or mbr with other utility functions) and in an mbrbased sensitivity analysis which compares the differences in mbr scores that arise when such errors are introduced to a candidate translation. we also show that this insensitivity is not simply the result of insufficient training data containing such errors: retraining comet with additional synthetic data did not fully alleviate this weakness. while errors related to number and named entity translation were very salient in our exploration, we do not claim that this case study is exhaustive. in our manual analysis, we also see anecdotal evidence of polarity errors and nonsensical german compounds. we hope our findings motivate further research into identifying and mitigating biases of neural machine translation metrics -we envision that actively searching for biases in neural metrics, for example by using them as utility functions in mbr, could become an important step during metric development.