Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation (2023)

Abstract: We propose a genetic algorithm (GA) based method for modifying n -best lists produced by a machine translation (MT) system. Our method offers an innovative approach to improving MT quality and identifying weaknesses in evaluation metrics. Using common GA operations (mutation and crossover) on a list of hypotheses in combination with a fitness function (an arbitrary MT metric), we obtain novel and diverse outputs with high metric scores. With a combination of multiple MT metrics as the fitness function, the proposed method leads to an increase in translation quality as measured by other held-out automatic metrics.With a single metric (including popular ones such as COMET) as the fitness function, we find blind spots and flaws in the metric. This allows for an automated search for adversarial examples in an arbitrary metric, without prior assumptions on the form of such example. As a demonstration of the method, we create datasets of adversarial examples and use them to show that reference-free COMET is substantially less robust than the reference-based version.

## introduction attaining good translation quality in machine translation (mt) arguably relies on good automatic metrics of mt quality. recently, a new generation of evaluation metrics was introduced. these metrics are based on embeddings computed by large pretrained language models and human annotation scores. the improvements in metric quality resulted in renewed interest in metric-driven translation hypothesis selection methods, like minimum bayes risk (mbr) decoding @xcite . our method relies on mbr decoding and the genetic algorithm (ga; @xcite @xcite through combinations and mutations of translations produced by an mt model, we search for optimal translation under a selected metric. this is a novel approach to generating translation hypotheses in nmt. we find that by combining neural and surface form-based metrics in a ga's fitness function, it is possible to create better quality translations than by simple reranking of the initial hypotheses (as evaluated by held-out metrics). it also allows the combination of multiple sources for the translation, for example, mt, paraphrasing models and dictionaries. another use-case for our method is the identification of weak points in mt metrics. flaws and biases of the novel neural metrics are being studied, for example, by @xcite , @xcite , @xcite or @xcite . in summary, these metrics have low sensitivity to errors in named entities and numbers. also, they are not sufficiently sensitive to changes in meaning and critical errors, like negations. these previous works on deficiencies of the metrics mostly focus on analyzing the outputs of mt systems and looking for certain types of mistakes. another approach they use is changing the outputs to introduce specific types of mistakes. in contrast, our approach aims to find translations with high scores on certain metrics automatically, by optimizing the candidate translations for a selected metric. we believe that through this more explorative approach, it is possible to find unexpected types of defects. in summary, the main contribution of our work is a novel method for producing translations, which can be used to improve translation quality and analyze automatic mt evaluation metrics. 1 2 related work automated mt evaluation the traditional automatic mt metrics are based on comparing a trans-lation produced by an mt system to a human reference based on a string similarity. popular choices are chrf @xcite and bleu @xcite . multiple shortcomings of these metrics are well known @xcite @xcite @xcite . neural mt metrics novel, neural-based mt metrics were introduced recently. they address some of the deficiencies of the string-based methods, but possibly introduce new types of errors or blind spots: bertscore @xcite , bartscore @xcite , prism (thompson and post, 2020), bleurt @xcite , comet @xcite @xcite , yisi @xcite , robleurt @xcite or unite @xcite . using a shared embedding space, these metrics better compare source, translated, and reference sentences. their evaluation in wmt metrics tasks @xcite @xcite and other campaigns @xcite demonstrate stronger agreement with human judgment. while their system-level performance has been scrutinized, their segment-level performance remains less explored. @xcite indicates these metrics are unreliable for assessing translation usefulness at segment level. however, we still try to optimize individual sentences for improved scores. deficiencies in metrics the closest work to ours is @xcite . authors use mbr decoding to find examples of high-scoring, but flawed translations in sampled model outputs. the conclusion is that the studied metrics are not sensitive to errors in numbers and in named entities (ne). @xcite automatically generate texts with various kinds of errors to test for sensitivity of mt metrics to such perturbations. @xcite claim that current mt quality estimation (qe) models do not address adequacy properly and @xcite further show that meaningchanging errors are hard to detect for qe. ## proposed solution our approach depends on two methods: minimum bayes risk decoding and genetic algorithm. ## experiments this section describes our experimental setup and results. we compare reranking of n-best lists to the application of the ga on them. ## analysis in this section, we analyze the ga procedure and the behavior of evaluation metrics. ## discussion we agree that an argument could be made that our approach is very computationally expensive, too explorative and the search for weaknesses could be performed in a more principled way. however, by anticipating the types of errors the metrics ignore and by designing the procedure to create texts with such errors, some of the error types can remain unnoticed. we see analogies with the whole field of deep learning. the methods with more priors of what the outcome should look like and how an inductive bias should be represented in a model give way to more general architectures as systems are scaled both in parameters and training data size, in the spirit of richard sutton's bitter lesson. 11 since the architectures of systems that produce evaluation scores are based mostly on empiric results, rather than on solid theoretical approaches, 11 http://www.incompleteideas.net/incideas/ bitterlesson.html we believe that similar empirical, almost bruteforce methods, might be an effective tool to search for weaknesses of these systems. ## conclusions we present a method of using a ga to find new translations based on optimizing hypotheses from an n-best list produced by an mt model. our method optimizes well towards an arbitrary mt metric through modification of the candidate translations. we found that after optimizing for a single objective metric, scores on other metrics often decrease, due to over-fitting on the objective metrics' defects. we discover that by combining multiple metrics (both neural and string-based) in the fitness (objective) function, we are able to mitigate the over-fitting and improve or maintain the held-out metrics for most inputs. this suggests ga can be used to improve mt quality. mt evaluation metrics have specific flaws and blind spots. to test their robustness, we selected some of the metrics as the fitness functions to optimize towards, and others as held-out metrics. we have leveraged the over-fitting effect to search for adversarial examples for specific metrics, creating translations that score high in one metric and low in held-out metrics. such translations can be used as negative examples for improving the robustness of the neural metrics. this work also reveals that even though source-translation and source-translation-reference comet scores were shown to have a similar correlation with human scores for well-formed translations, the reference-free comet is more susceptible to adversarial inputs.this highlights the necessity of thorough analysis, beyond computing correlation with human scores for the new metrics.