BLEU Meets COMET : Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation (2023)

Abstract: Although neural-based machine translation evaluation metrics, such as COMET or BLEURT, have achieved strong correlations with human judgements, they are sometimes unreliable in detecting certain phenomena that can be considered as critical errors, such as deviations in entities and numbers. In contrast, traditional evaluation metrics such as BLEU or chrF, which measure lexical or character overlap between translation hypotheses and human references, have lower correlations with human judgements but are sensitive to such deviations. In this paper, we investigate several ways of combining the two approaches in order to increase robustness of state-of-the-art evaluation methods to translations with critical errors. We show that by using additional information during training, such as sentence-level features and word-level tags, the trained metrics improve their capability to penalize translations with specific troublesome phenomena, which leads to gains in correlations with humans and on the recent DEMETR benchmark on several language pairs.

## introduction trainable machine translation (mt) evaluation models, such as comet @xcite and bleurt @xcite , generally achieve higher correlations with human judgments, thanks © 2023 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. 1 our code and data are available at: https://github.com/ deep-spin/robust mt evaluation to leveraging pretrained language models. however, they often fail at detecting certain types of errors and deviations from the source, for example related to translations of numbers and entities @xcite . as a result, their quality predictions are sometimes hard to interpret and not always trustworthy. in contrast, traditional lexical-based metrics, such as bleu @xcite or chrf @xcite -despite their many limitations-are considerably more sensitive to these errors, due to their nature, and are also more interpretable, since the scores can be traced back to the character or n-gram overlap. this paper investigates and compares methods that combine the strengths of neural-based and lexical approaches, both at the sentence level and at the word level. this is motivated by the findings of previous works, which demonstrate in detail that the comet mt evaluation metric struggles to handle errors like deviation in numbers, wrong named entities in generated translations, deletions that exclude important content from the source sentence, insertions of extra words that are not present in the source sentences, and a few others @xcite . while data augmentation techniques alleviate the problem to some extent @xcite , the gains seem to be relatively modest. in this paper we investigate alternative methods that take advantage of lexical information and go beyond the use of various augmentation techniques and synthetic data. we focus on increasing robustness of mt evaluation systems to certain types of critical errors. we experiment with the reference-based comet metric, which has access to reference translations when producing quality scores. in order to make evaluation metrics more robust towards this type of errors, we consider and compare three different ways of incorporating information from lexical-based evaluation metrics into the neural-based comet metric: we compare these three strategies with the recent approach of @xcite , which generates synthetic data with injected errors from a large language model, and retrains comet on training data that has been augmented with these examples. we assess both the correlation with human judgments and using the recently proposed demetr benchmark @xcite . ## related work recently several challenge sets have been introduced, either within a scope of the wmt metrics shared task @xcite or in general as a step towards implementing more reliable mt evaluation metrics: smaug @xcite explores sentence-level multilingual data augmentation; aces @xcite is a translation accuracy challenge set that covers high number of different phenomena and language pairs, including a considerable number of low-resource ones; demetr @xcite and hwtsc @xcite aim at examining metrics ability to handle synonyms and to discern critical errors in translations; dfki @xcite employs a linguistically motivated challenge set for two language directions (german ↔ english). apart from purely focusing on improving robustness with augmentation of different phenomena, there are works that combine usage of synthetic data with other different methods. these methods use more fine-grained information-aiming at identifying both the position and the type of translation errors on given source-hypothesis sentence pairs @xcite . as another source of useful information, word-level supervision can be considered, which has proven to be beneficial in tasks of quality estimation and mt evaluation @xcite . there have been other attempts to add linguistic features to automatic mt evaluation metrics, e.g. incorporating information from a multilingual knowledge graph into bertscore. @xcite proposed a metric that linearly combines the results of bertscore and bilingual named entity matching for reference-free machine translation evaluation. @xcite use an extensive set of linguistic features at word-and sentence-level to aid sentiment classification. additionally, glassbox features extracted from the mt model have been used successfully in the quality estimation task @xcite @xcite . for the incorporation of different types of information to neural models early and late fusion is commonly used with benefits on multiple tasks and domains @xcite @xcite . to the best of our knowledge there have not been any attempts to combine the representations of neural metrics with external features obtained by lexical-based metrics. moreover, there are similar concerns regarding robustness of evaluation models in non-mt related tasks (chen and eger, 2022). in general, it is depicted that evaluation metrics perform rather well on standard evaluation benchmarks but are vulnerable and unstable to adversarial examples. the approaches investigated in our paper aim to address these limitations. ## combination of neural and lexical metrics in this section we describe the methods we investigated in order to infuse the comet with information on lexical alignments between the mt hypothesis and the reference. ## experimental design the main focus of our experiments is to investigate how the robustness of the mt evaluation models can be improved and how the proposed settings compare to each other and to a data augmentation approach proposed by @xcite . our comparisons address the correlation with human judgments and recent robustness benchmarks on mt evaluation datasets ( §4). we follow (amrhein and sennrich, 2022) -we use comet (v1.0) @xcite as the underlying architecture for our mt evaluation models and focus on making it more robust. human judgements data we consider two types of human judgments: direct assessments (da) @xcite and multi-dimensional quality metric scores (mqm) @xcite . for training, we use wmt 2017-2020 data from the metrics shared task @xcite with direct assessment (da) annotations (see app. c). for development and test, we use the mqm annotations of the wmt 2021 and 2022 datasets, respectively foot_1 . challenge sets data furthermore, we evaluate our models using two challenge sets: demetr @xcite and aces @xcite . figure 2 : the architecture of the comet model with incorporated word-level lexical features. both of these challenge sets allow measuring the sensitivity of the proposed approaches to various phenomena and assess their overall robustness. augmentation we compare our methods against the multilingual data augmentation approach smaug foot_2 proposed by @xcite . specif-ically, we use transformations that focus on deviations in named entities and numbers since these are identified as the major weaknesses of comet @xcite . ## results and discussion in this section, we show results for the aforementioned methods, specifically the correlations with mqm annotations from wmt 2022 metrics shared task for 3 high-resource language pairs (@xmath0, @xmath1, @xmath2: conversation, e-commerce, news and social. in addition, we discuss evaluation results obtained on two challenge sets. ## conclusion and future work in this paper, we presented several approaches that use interpretable string-based metrics to improve the robustness of recent neural-based metrics such as comet. there are various ways of combining these methods together: ensembling metrics, incorporating sentence-level features, or using wordlevel information coming from alignments between the hypothesis and the reference. we observed that adding small changes to the architecture of comet, either by using sentence-level features based on bleu and chrf scores, or by incorporating wordlevel tags for the hypothesis, can lead to competitive performance gains. to showcase the effectiveness of our proposed approaches, we evaluated them on the most recent mqm test set that covers multiple domains and language pairs, as well as on the challenge sets that were introduced in the wmt 2022 metrics shared task, with encouraging results. it is likely that our proposed approaches are complementary to each other, as well as to the data augmentation method we are comparing against (comet+aug). an interesting direction for future work is to study further the impact of using word-level tags of the hypothesis in other ways not covered in this paper, e.g., in combination with augmentation approaches.