T 5 S core: Discriminative Fine-tuning of Generative Evaluation Metrics (2023)

Abstract: Modern embedding-based metrics for evaluation of generated text generally fall into one of two paradigms: discriminative metrics that are trained to directly predict which outputs are of higher quality according to supervised human annotations, and generative metrics that are trained to evaluate text based on the probabilities of a generative model. Both have their advantages; discriminative metrics are able to directly optimize for the problem of distinguishing between good and bad outputs, while generative metrics can be trained using abundant raw text. In this paper, we present a framework that combines the best of both worlds, using both supervised and unsupervised signals from whatever data we have available. We operationalize this idea by training T5Score, a metric that uses these training signals with mT5 as backbone. We perform an extensive empirical comparison with other existing metrics on 5 datasets, 19 languages and 280 systems, demonstrating the utility of our method. Experimental results show that: T5Score achieves the best performance on all datasets against existing top-scoring metrics at the segment level.

## introduction automatically evaluating the quality of generated text plays an essential role in the development of text generation systems @xcite @xcite . a key element of this evaluation is the design of an automated metric that can recognize high-quality texts. the current most-popular approach to create such high-quality metrics is the discriminative paradigm. these models are generally trained by taking a sentence embedding model and fine-tuning it using human judgments of generated text quality as a learning signal, allowing metrics to directly predict the quality score of a text. popular examples include comet @xcite and bleurt @xcite . however, the effectiveness of this method comes at the cost of expensive manual annotation of human judgements, and thus these models are less broadly applicable than more common lexical metrics such as bleu @xcite , rouge @xcite . more recently, there has been promising work on generative metrics. these metrics recognize high-quality texts by formulating evaluation as a generation task and using the generative likelihood as an indication of quality, making it possible to train the models without explicit human annotation. examples of these metrics include bartscore @xcite and prism @xcite . however, because such generative models do not utilize human judgements at training time, these models are inherently at a disadvantage compared to metrics that can utilize supervision. in this work, we argue that it is crucial to utilize all possible supervision symbols that could indicate the quality of the text. to this end, we propose a framework for learning evaluation metrics based on the assumption that generative and discriminative objectives can work in concert to train a better evaluator, as shown in fig. 1 . we achieve this idea by (1) starting with the pre-trained model mt5 @xcite , (2) training mt5 in a generative fashion by maximizing the probability of existing parallel data, then (3) fine-tuning mt5 discriminatively by minimizing a contrastive loss function to teach it to ensure that the generative probability of high-quality texts is higher than that of low-quality texts. at evaluation time, the probability of generating a text is used as the quality score, because the model has learned to assign high probability to superior texts. our framework has the flexibility to choose from a supervised training strategy and an unsupervised training strategy depending on if human judgements are available for a given language or task, while keeping the evaluation process the same. we evaluate the proposed metric (t5score) on 5 datasets covering machine translation (mt) and summarization tasks across 19 languages. regarding reference-based experiments, at the segment level, t5score trained generatively achieves the best performance on one dataset without human annotated training examples; t5score trained discriminatively achieves the best performance on 4 datasets with human annotated training examples against top-scoring counterparts. at the system level, t5score trained discriminatively achieves the best performance in 5 of 6 test settings (2 correlation methods × 3 datasets). empirical results also show the effectiveness of generative training, especially for tasks without human judgments. regarding source-based experiments, we find it better at evaluating top-scoring systems compared to reference-based evaluation, showing the importance of developing source-based evaluation as ma-chines generate higher-quality texts. ## task formulation text generation evaluation aims to design a function auto_eval(•) that takes in a source text x, some reference outputs y and a system output ŷ and predicts a scalar value that indicates the quality of the system output. the validity of the designed function depends on the degree of correlation between auto_eval(•) and human judgements (which can be denoted as manual_eval(•)). the better the correlation between the two, the more effective we consider our designed function to be. specifically, in this work, we call an evaluation function (1) source-based if it takes only x and ŷ and predicts using auto_eval(x, ŷ) (2) and call an evaluation function reference-based if it takes only y and ŷ and predicts using auto_eval(y, ŷ), or it takes x, y and ŷ and predicts using auto_eval(x, y, ŷ). ## metric design in this section, we describe t5score and explain how to train the metric in both a generative and discriminative fashion. ## reference-based evaluation we consider two tasks in reference-based evaluation: machine translation (da20, mqm20 and mqm21) and summarization (multisumm). ## source-based evaluation we also support source-based discriminatively trained methods. in this section, we show the effectiveness of the source-based method. we consider the task of machine translation and inspect the re- ## analysis we design analyses to better understand the mechanism of t5score and its strenghs over other metrics, specifically asking three questions: q1: is generative training necessary before discriminative training? q2: what are the strengths and weaknesses of each evaluation metric? q3: when will source-based evaluation outperforms reference-based evaluation? effectiveness of generative training in our experiments, discriminative training is based on the model trained generatively. to answer q1, we compare the performance of discriminatively trained models with and without generative training on da20. the results are shown in fig. 4 . 15 we observe that (1) both t5score-b and t5score-l are enhanced by generative training before discriminative training under three correlation metrics (except that t5score-b using segment level kendall's tau correlation has a little performance drop), which means that generative training is necessary to improve model performance. (2) larger model t5score-l benefits more from generative training, compared to t5score-b, indicating that larger models are better at keeping knowledge from generative training. seg-k sys-p sys-k 40 60 80 correlation t5score-b w/o t5score-bw t5score-l w/o t5score-lw ## related work the increasing performance of generation systems equipped with large pre-trained models puts forward a higher requirement of the evaluation ability of automated metrics. as such, researchers are exploring different evaluation frameworks by teaching metric to learn diverse types of knowledge. the directest way is to supervise metrics with manually annotated judgments with a trainable model, typical works include beer @xcite , bleurt @xcite , robleurt @xcite , comet @xcite and c-spec @xcite . despite the superior performance of optimizing the correlation with human judgments, this method is expensive in creating human annotations. to bypass this challenge, researchers attempt to evaluate generated texts in an unsupervised way by calculat-ing the lexical or semantic similarity between reference and generated texts with surface-based string match (e.g., bleu @xcite , rouge @xcite and chrf @xcite ) or unsupervised pre-trained components (e.g., bertscore @xcite , yisi @xcite and moverscore @xcite ). recently, works such as prism @xcite and bartscore @xcite start to formulate evaluation as a generation task, which can not only make full use of pre-trained knowledge but also find more training data that can provide useful supervision for metrics to learn. in this paper, we propose a framework that allows different types of signals to be incorporated into metrics. concurrent with our work, unieval @xcite formulates evaluation as a boolean question answering task and is trained on semi-supervised data. by contrast, our method is based on the generation formulation, which enables us to utilize a large amount of raw parallel data. ## conclusions in this paper, we augment evaluation metrics with the ability to use different types of signal from data, which based on the assumption that a good evaluation metric not only should be informed of how to score different qualities of texts but also how highquality texts are generated. we achieve this goal by proposing a discriminative generation frame-work for the evaluation of generated texts, which outperform 8 existing top-performing metrics on 5 datasets from 19 languages.