Robust MT Evaluation with Sentence-level Multilingual Augmentation (2022)

Abstract: Automatic translations with critical errors may lead to misinterpretations and pose several risks for the user. As such, it is important that Machine Translation (MT) Evaluation systems are robust to these errors in order to increase the reliability and safety of Machine Translation systems. Here we introduce SMAUG a novel Sentence-level Multilingual AUGmentation approach for generating translations with critical errors and apply this approach to create a test set to evaluate the robustness of MT metrics to these errors. We show that current State-of-the-Art metrics are improving their capability to distinguish translations with and without critical errors and to penalize the first accordingly. We also show that metrics tend to struggle with errors related to named entities and numbers and that there is a high variance in the robustness of current methods to translations with critical errors.

## introduction in recent years, machine translation (mt) systems have been used in diverse real world environments. however, widespread adoption of these systems raises many concerns, namely in the quality of their outputs. ideally, human translators would evaluate generated translations but this process is expensive and slow. as an alternative, automatic machine translation evaluation relies on external systems to measure the quality of generated translations. as a crucial aspect of machine translation evaluation, it is vital to ensure that generated sentences do not contain critical errors. as detailed in @xcite , translations with such errors deviate in meaning from their source sentence in ways that may lead to misinterpretations and pose health, safety, legal, reputation, religious or financial implications. @xcite group these translations into three categories, based on how their * corresponding author: duartemalves@tecnico.ulisboa.pt meaning deviates from the source sentence. mistranslation errors have critical content in the source sentence translated into a different meaning, not translated (the content remains in the source language), or translated into gibberish. hallucination errors introduce content in the translated sentence that is not present in the source sentence. deletion errors exclude important content from the source sentence. in this work, we propose smaug foot_0 , a sentencelevel multilingual augmentation framework to generate translations with critical errors, targeting all the aforementioned critical error categories. we also introduce a novel test set to analyse the robustness of mt evaluation systems to critical errors. this test set was created with the proposed augmentation framework and submitted to the wmt22 challenge sets sub-task @xcite . finally, we present the results obtained from evaluating metrics submitted to the wmt22 metrics shared task with the developed test set. we show progress of submitted metrics with respect to baseline systems, particularly concerning quality estimation systems. namely, we demonstrate that several metrics are able to correctly distinguish translations with and without critical errors and to penalize the former. furthermore, we show that current metrics are less sensitive to translations containing errors in named entities and numbers and that there is a high variance in the performance of current sota evaluation metrics with respect to identifying and penalizing the occurrence of critical errors. ## related work metrics for machine translation evaluation produce a quality score for a given hypothesis, based on the source sentence and a possibly empty set of reference translations. these metrics can be divided into two main groups, given their reference set. reference based metrics have a non-empty reference set, while reference free metrics have an empty reference set. reference free evaluation is also denominated by quality estimation. within reference-based metrics, n-gram based metrics, such as bleu @xcite and chrf @xcite , measure lexical overlap from the hypothesis to the human references. @xcite advocate that these methods fail to capture semantic similarities beyond the lexical level. their inability to capture meaning at a sentence level also makes them unfit for the detection of critical errors as they equally penalize the usage of synonyms or the mistranslation of a named entity. as an alternative to n-gram matching, more recent methods leverage word representations to capture semantic similarities beyond the lexical level. as described in @xcite , embedding-similarity methods, like yisi-1 @xcite and bertscore @xcite , create an alignment between the vector representations of the words in the hypothesis and the reference and then compute a score that captures the semantic similarity between both sentences. as noted by @xcite , the main issue with these approaches is that human judgements consider other information beyond semantic similarity, limiting the correlation of these methods with human evaluations. more recently, learnt methods, such as bleurt20 @xcite and comet @xcite , address this issue by training to directly maximize correlation with human judgements. results from the wmt21 metrics @xcite and the wmt21 quality estimation @xcite shared tasks suggest that these methods obtain higher correlations with human judgements, such as direct assessments @xcite , human translation edit rate (hter) @xcite or multi-dimensional quality metrics (mqm) @xcite . however, as noted by @xcite , relying on accuracy on held-out sets can lead to an overestimation on the performance of nlp models. as such, @xcite proposes checklist that relies on data augmentation techniques to create examples that test specific behaviours of nlp systems in various situations. within the field of machine translation evaluation, as a case study for exploring the sensitivity of learnt metrics to specific phenomena, @xcite employed minimum bayes risk decoding with comet as an utility function to identify good hypotheses. the authors show that hypotheses chosen with comet are more likely to have errors in named entities and numbers when compared to chrf, indicating the metric is not sensitive enough to these errors. considering multiple metrics, @xcite tested multiple systems on a challenge set with errors related to negation and sentiment polarity and found that most metrics struggle with these errors. nonetheless, these examples were chosen from existing mt outputs, which can lead to a major human effort, as these errors are not common. regarding reference free evaluation, @xcite define multiple perturbations to test the robustness of qe systems in detecting specific errors. the authors show that overall the tested perturbations are well detected but some, such as polarity based perturbations, still pose a challenge to qe systems. however, the list of perturbations is not exhaustive and most rely on transformations that do not necessarily preserve the semantics of the phrases, such as random insertions, substitutions and deletions. ## smaug framework in order to create an example of a critical error, the proposed framework receives an existing sentence and perturbs it, inducing one of the linguistic phenomena detailed in the following sections. for each linguistic phenomenon, the perturbation process is separated into two phases: transformation and validation. the first phase generates a candidate sentence by perturbing the original translation. this phase may not produce a candidate, as some perturbations are not applicable to all sentences. the second phase verifies whether the produced candidate meets a set of desirable criteria, discarding it otherwise. ## challenge set the created test set comprises of records in the format (s, h good , h bad , r, p), where s is a source sentence, h good and h bad are "good" and "bad" hypothesis, r is a reference and p is an identifier for the linguistic phenomenon present in h bad . three language pairs were considered: english-portuguese, spanish-english, portuguese-english. for each language pair, a data augmentation approach was applied to an existing parallel corpus to generate a the final set of records. ## experiments the developed test set was submitted to the wmt22 challenge set sub-task and the scores for several state-of-the-art metrics were gathered. the following sections detail the evaluation method for the tested metrics and the obtained results. ## conclusions ensuring generated translations do not have critical errors is a crucial aspect of machine translation evaluation, as they can pose various risks. in this work, we propose smaug, a multilingual augmentation framework to create translations with critical errors by inducing several linguistic phenomena in existing translations. we also apply these perturbations to create a manually verified test set to assess the robustness of machine translation evaluation systems to critical errors. with the created test set, we evaluate multiple metrics and show promising progress in current state-of-the-art methods in both distinguishing translations with and without critical errors and significantly penalizing the occurrence of critical errors in translations. nevertheless, errors related to named entities and numbers were found to pose a challenge for several tested metrics. additionally, we observe a high variance in the measured correlations across all the developed phenomena, suggesting an unpredictability on the performance of current methods with respect to detecting critical errors. one of the challenges in the automatic generation of translations with critical errors is the validation of the output. in this work, we relied on a preliminary automatic validation but also required a manual verification of the outputs. future work will explore high-precision validation techniques, such as the work of @xcite that uses very specific detectors to find examples of critical errors in translations. furthermore, support for multiple languages is a crucial aspect of this framework. however, several of the devised perturbations support a limited number of languages pairs. for example, the deviation in meaning phenomenon only supports to-english language pairs, as the polyjuice model is an english only model. a future avenue of research will investigate methods to expanding the number of languages supported by the linguistic phenomena.