Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks (2023)

Abstract: We investigate MT evaluation metric performance on adversarially-synthesized texts, to shed light on metric robustness. We experiment with word- and character-level attacks on three popular machine translation metrics: BERTScore, BLEURT, and COMET. Our human experiments validate that automatic metrics tend to overpenalize adversarially-degraded translations. We also identify inconsistencies in BERTScore ratings, where it judges the original sentence and the adversarially-degraded one as similar, while judging the degraded translation as notably worse than the original with respect to the reference. We identify patterns of brittleness that motivate more robust metric development.

## introduction automatic evaluation metrics are a key tool in modern-day machine translation (mt) as a quick and inexpensive proxy for human judgements. the most common and direct means to evaluate an automatic metric is to test its correlation with human judgements on outputs of mt systems. however, as such metrics are commonly used to inform the development of new mt systems and even used as training and decoding objectives @xcite , it is inevitable for them to be applied to out-of-distribution texts that do not frequently occur in existing system outputs. the rapid advancement of mt systems and metrics, as well as the prospect of incorporating mt metrics in the training and generation process, motivates investigation into mt metric robustness. in this work, we examine textual adversarial attacks (taas) as a means to synthesize challenging translation hypotheses where automatic metrics systematically underperform. we experiment with word- @xcite @xcite and character-level @xcite attacks on three popular, high-performing automatic mt metrics: bertscore @xcite , bleurt @xcite , and comet @xcite . we construct situations where the metrics disproportionately penalize adversariallydegraded translations. to validate such situations, we collect a large set of human ratings on both original and adversarially-degraded translations. as bertscore can also be seen as a measure of semantic similarity between any two sentences, we also explore another scenario of inconsistency where bertscore judges the original and adversarially-perturbed translations as similar while judging the perturbed translation as notably worse than the original one with regard to the reference translation. examples are shown in figure 1 . we identify mask-filling and word substitution as effective means to generate perturbed translations where bertscore, bleurt, and comet over-penalize degraded translations and bertscore is self-inconsistent. in particular, bleurt and comet are more susceptible to perturbations in data with higher-quality translations. our findings serve as a basis for developing more robust automatic mt metrics. foot_0 2 methods ## related work classical mt metrics such as bleu @xcite and rouge @xcite have been shown to correlate poorly with human judgements @xcite , motivating the development of model-based metrics. supervised metrics such as bleurt @xcite and comet @xcite are trained to mimic human ratings as a regression task. nonlearnable metrics such as bertscore @xcite , xmoverdistance @xcite , and uscore @xcite do not rely on human ratings and instead leverage the embeddings of the source, reference, and translation. whereas more recent and higher-performing metrics exist, we focus our investigation on bertscore, bleurt, and comet as they are most commonly used and easily adapted to other domains such as text simplification @xcite . @xcite defines robustness as performance on unseen test distributions. such distributions can occur naturally @xcite or be constructed adversarially, and robustness failures are usually identified through human priors and error analysis. @xcite and @xcite use hand-crafted types of perturbations to create challenge sets where mt metrics underpenalize the perturbed translations. @xcite use minumum risk training @xcite to optimize directly for higher metric scores, resulting in a set of translations with overestimated scores. this work complements previous works by investigating overpenalization, which is a very different behavior to overestimation. we use adversarial attacks targeted at each metric that are not limited to pre-defined categories, which allows us to discover particular failure cases specific to each metric. in addition, we consider the same set of metrics on different years of wmt data. this allows us to draw connections between adversarial robustness and the quality of mt system outputs, and whether the mt system outputs are used when training the metric. ## conclusion we apply word-and character-level adversarial attacks and probe for overpenalization with bertscore, bleurt, and comet, and for selfinconsistencies with bertscore. we observe that mask-filling and word substitution are more effective at generativing challenging cases, and that bleurt and comet are more susceptible to perturbation of high-quality translations. our findings motivate more sophisticated data augmentation and training methods to achieve greater metric robustness. in particular, our formulation of self-consistency requires no validation against human ratings and can be applied to other embedding-based metrics @xcite @xcite as a regularization term. we leave this to future work.