M etric X -23: The G oogle Submission to the WMT 2023 Metrics Shared Task (2023)

Abstract: This report details the MetricX-23 submission to the WMT23 Metrics Shared Task and provides an overview of the experiments that informed which metrics were submitted. Our 3 submissions—each with a quality estimation (or reference-free) version—are all learned regression-based metrics that vary in the data used for training and which pretrained language model was used for initialization. We report results related to understanding (1) which supervised training data to use, (2) the impact of how the training labels are normalized, (3) the amount of synthetic training data to use, (4) how metric performance is related to model size, and (5) the effect of initializing the metrics with different pretrained language models. The most successful training recipe for MetricX employs two-stage fine-tuning on DA and MQM ratings, and includes synthetic training data. Finally, one important takeaway from our extensive experiments is that optimizing for both segment- and system-level performance at the same time is a challenging task.

## introduction automatic evaluation metrics are critical to the development of machine translation (mt) systems. they are the most frequently used method for comparing two mt systems and deciding which generates higher quality translations. each year, the conference on machine translation (wmt) runs a metrics shared task to benchmark the quality of state-of-the-art evaluation metrics @xcite . meta-evaluating metrics by measuring how well they correlate to human ratings of translation quality is critical for understanding the extent to which automatic evaluations of mt systems are trustworthy. this report details the metricx-23 submission to the metrics shared task. metricx is a learned 1 our code and mt5-based models can be found at https: //github.com/google-research/metricx . metricx-23 mt5-xxl da 15-20, z-normalization, scores clipped to [-1, 1] mqm 20-21 + synthetic data, raw mqm scores metricx-23-b mt5-xxl da 15-20, z-normalization, scores clipped to [-1, 1] mqm 20-22 + synthetic data, raw mqm scores metricx-23-c palm 2 (bison) da 15-20, z-normalization, scores clipped to [-1, 1] mqm 20-21 + synthetic data, raw mqm scores metricx-23-b and metricx-23-c differ from the primary submission in that the "b" version is finetuned on mqm 2022 data in addition to 2020 and 2021, and the "c" version uses palm 2 as its pretrained language model (differences in bold). each of the submission also includes a qe variant that follows the same training procedure. ## metric descriptions the metricx-23 metrics that were submitted to the metrics shared task are all learned regressionbased metrics that are trained to predict a floating point number that represents the quality of a given translation. the input to the reference-based metrics is the candidate translation (hypothesis) and reference segments-each with a corresponding prefix ("candidate:" and "reference:", respectively)concatenated together. the combined input is encoded by the model, and then the metric uses the encoding to predict a score. this stands in contrast to comet-style metrics in which the hypothesis and reference are encoded separately, then combined in order to predict a score @xcite . the qe variants use the source segment instead of the reference, with the prefix changed to "source:". we use two different network architectures for different versions of the metric. the choice of architecture depends on which pretrained language model is used to initialize the model. the first architecture is based on the encoderdecoder mt5 language model @xcite . the input is encoded by the encoder, then the output logit from an arbitrary token in the vocabulary distribution from the first step of decoding is selected to represent the score for the hypothesis and trained accordingly. 2 in practice, we found that this method for using the pretrained weights for both the encoder and decoder worked better than using a regression head on top of the encoder and discarding the decoder. the second architecture is the prefix language model based on transformer @xcite used by the palm 2 model @xcite . we augment the architecture by adding a feedforward regression layer on top of the input encoding. the output from the feedforward layer is trained to predict the translation quality score. both types of model are trained with a mean squared error (mse) loss function. further implementation details related to checkpoint selection, optimization, etc., can be found in §3.3. information related to training data, label normalization, etc., can be found in §4. ## experimental results we made three different submissions to the shared task, each with a reference-based and qe variant: an overview of these submissions is shown in figure 1. in the rest of this section, we describe the experimental results that led us to these submissions. the experiments in the process of determining the best training recipe were performed with mt5-xl (3.7b parameters), but our final submissions then use the xxl variant with 13b parameters. all results are reported as the mean of 3 independent runs, along with the standard deviation across the runs, unless stated otherwise. ## related work for many years, lexical-based metrics like bleu @xcite and chrf @xcite were the standard method for automatically evaluating mt output. however, as it was demonstrated that learned evaluation metrics correlate to human ratings significantly higher than lexical-based metrics @xcite , the vast majority of recent research on mt evaluation has used learned metrics. more recent approaches to mt evaluation leverage large language models (llms) to do zero-shot scoring by either directly predicting scalar quality scores or phrase-level error tagging @xcite . these approaches typically leverage models that are orders of magnitude larger than metrics that are trained specifically for mt evaluation. in comparison to the metricx submission to the wmt22 metrics shared task, this year's submission shares the same architecture, but we performed a significantly larger number of experiments to arrive at the final models, which are detailed in this report. we also explore how metric performance changes as a function of the number of parameters, experiment with initializing with different pretrained language models, and include a qe submission. ## conclusion in this report, we presented in detail our approach to training metricx-23, a regression-based mt evaluation metric. we submitted six versions of metricx-23 to the wmt23 metrics shared task, including both reference-based and qe variants. some of our findings are that (1) training on direct assessment (da) ratings and subsequently on mqm ratings leads to a significantly better performance than training on either of the two datasets alone, (2) z-normalization of da ratings helps achieve better segment-level performance, but is not useful for high-quality mqm ratings, and (3) adding a small amount of synthetic data to the training set, targeting a challenge set, can also boost the metric's overall performance. throughout our experiments, we observed an undesirable tension between segment-and systemlevel performance, making it challenging to improve our metric in both aspects at the same time. nevertheless, increasing the size of the model used to pre-initialize the metric appears to be one reliable way to increase the overall performance, at least to a certain point. future work may benefit from a better understanding of the trade-off between segment-and system-level performance, and whether it would be better to focus on separate metrics for these two types of mt evaluation.