BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training (2023)

Abstract: Automatic metrics play a crucial role in machine translation. Despite the widespread use of n-gram-based metrics, there has been a recent surge in the development of pre-trained model-based metrics that focus on measuring sentence semantics. However, these neural metrics, while achieving higher correlations with human evaluations, are often considered to be black boxes with potential biases that are difficult to detect. In this study, we systematically analyze and compare various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. Through Minimum Risk Training (MRT), we find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm. By incorporating token-level constraints, we enhance the robustness of evaluation metrics, which in turn leads to an improvement in the performance of machine translation systems. Codes are available at https://github.com/powerpuffpomelo/fairseq_mrt .

## introduction automatic metrics are crucial for the training of machine translation models, as they can measure translation quality at low cost. currently, the most widely used translation evaluation metric is still the n-gram-based bleu @xcite . however, it is acknowledged that bleu, which relies on the surface-level vocabulary matching, exhibits significant limitations @xcite @xcite . for instance, bleu fails to differentiate between errors of varying severity and assigns equal weight to each word. in recent years, the advent of pre-trained models @xcite @xcite @xcite has led to significant advancements in the development of metrics such as bleurt @xcite and comet @xcite , which employ pretrained language models (plm) to assess the semantic meaning of sentences. these approaches have been shown to outperform metrics that rely on superficial word matching and have a more consistent correlation with human annotation. despite these advances, it is important to note that neural metrics are characterized by opaque decision bases and may be subject to biases that are more difficult to detect @xcite . therefore, we aim to conduct an analysis of the properties of various metrics in order to gain a deeper understanding. while there have been recent studies on the analysis of metrics @xcite @xcite , these works primarily focus on examining metric scores on specific datasets. to the best of our knowledge, this paper is the first to analyze metrics from the perspective of their guidance for training machine translation systems. in this paper, we employ minimum risk training (mrt) @xcite models. compared to maximum likelihood estimation (mle), mrt can reduce the gap between training and evaluation, resulting in higher quality translations @xcite . in addition, since mrt uses metrics to optimize translation models, we can explore the impact of metrics on translation by observing the mrt training process. our experiment results show that mrt reveals the robustness defects in some metrics: the training collapses and the generated translations, despite getting high metric scores, show poor translation quality. for instance, we find universal adversarial translations of bleurt and bartscore, which are capable of obtaining high scores when evaluated against any reference sentence. an example is presented in figure 1 . further analysis shows that the robustness defects are rooted in the distribution biases of the training corpora, as well as in the tendency of the metric modeling paradigm. in addition, we explore methods for optimizing metrics and translation models: word-level information constraints are introduced by combining mrt with nll loss and metric ensemble. our main contributions are as follows: ## analyze metrics with mrt we train translation models in two stages: in the mle training phase, the model is trained with con-ventional negative log-likelihood (nll) loss; then in the mrt training phase, we fine-tune the model with each metric, so as to obtain translation models with various metric styles. in this way, the characteristics of different metrics can be analyzed through observing the changes in the training process and the translation results. ## optimize metrics and translations the analysis of the mrt training process allows us to understand the impact of each metric on translation quality. our goal is both to exploit the advantages of the mrt training approach and to avoid training collapse due to the robustness deficiencies of the metrics. mrt needs to sample many translation sentences in advance, and then use sentence-level metrics to predict the scores and calculate the loss. if the metrics that guide translation training do not take word-level information into account, the translation model may ignore details and gradually deviate during the training process. therefore, we try two methods to constrain the training direction by introducing word-level constraints: combining mrt and nll loss, and doing metrics ensemble. figure 4 : this figure displays two improvement strategies (<@xmath0, with each color representing a different training approach. the figure is divided into seven groups from left to right, representing the range of change in each metric after training with a particular approach. for instance, the green bar represents the impact of utilizing bleurt to guide translation model training in mrt, where only bleurt improves while the other metrics decline significantly (we only display the range of -20% to 20% ), which is consistent with the previous charts. ## conclusion in this paper, we present the first systematic analysis of automatic metrics from the perspective of guidance for training machine translation systems. we find that mrt reveals the robustness deficiencies of some metrics, such as universal adversarial translations of bleurt and bartscore, and we further analyze the underlying causes. in addition, we explore methods to improve metric robustness, thus helping to further enhance the performance of translation systems.