F rugal S core: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation (2022)

Abstract: Fast and reliable evaluation metrics are key to R&D progress. While traditional natural language generation metrics are fast, they are not very reliable. Conversely, new metrics based on large pretrained language models are much more reliable, but require significant computational resources. In this paper, we propose FrugalScore, an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance. Experiments with BERTScore and MoverScore on summarization and translation show that FrugalScore is on par with the original metrics (and sometimes better), while having several orders of magnitude less parameters and running several times faster. On average over all learned metrics, tasks, and variants, FrugalScore retains 96.8% of the performance, runs 24 times faster, and has 35 times less parameters than the original metrics. We make our trained metrics publicly available, to benefit the entire NLP community and in particular researchers and practitioners with limited resources.

## introduction automatic evaluation metrics are the only way to monitor the training of, evaluate, and compare across models in a systematic, large-scale way, and are thus a critical component of the research and development ecosystem in machine learning. to get adopted in practice, evaluation metrics need to be both reliable and affordable, i.e., fast and easy to compute. while some metrics meet these criteria, such as precision and recall in information retrieval, root mean square error in regression, etc., finding suitable metrics is still an open problem in the field of natural language generation (nlg) @xcite . following the advent of transfer learning in nlp, new nlg metrics based on large pretrained language models have recently been proposed, such as bertscore @xcite and moverscore @xcite . by relying on contextual embeddings, these metrics capture semantics and are therefore much more reliable. however, due to the sheer size of the underlying models, these metrics pose environmental issues @xcite , take time to compute, and require access to significant computational resources, so they are not accessible by everyone in the nlp community. for example, we were not able to run some of the best variants of bertscore foot_0 , based on deberta-large and deberta-xlarge @xcite on a 12gb gpu. even when enough gpu memory is available, relying on such large models is still associated with extended runtimes, which can impede the progress of experiments when used once or more per epoch for validation and monitoring purposes. to address this problem, we propose in this paper frugalscore, an approach to learn a lightweight version of bertscore, moverscore, and more generally any metric based on a large pretrained language model. our contributions can be summarized as follows: 1) our compact models have several orders of magnitude less parameters than the original metrics and run several times faster, while retaining most of their original performance. we even outperform the original metrics in some cases 3 . 2) our metrics are not only faster because of the much smaller amount of parameters, but also because they do not rely on any similarity function. ## background related work falls into two categories: unsupervised and supervised metrics. ## our approach developing frugalscore requires three phases, one of which is optional. phase 1. we create a synthetic dataset (see subsection 3.1) by sampling pairs of more or less related sequences and annotating them with the expensive metrics to be learned. this is a one-time operation that does not need to be repeated regardless of the model used in phase 2. phase 2. we continue the pretraining (see subsection 3.2) of a miniature pretrained language model on the synthetic dataset built by phase 1. here, the miniature model learns the internal mapping of the expensive metric, including any similarity function applied to the representations. note that a different miniature is trained for each metric to be learned (we leave learning metric combinations as future work). the miniature can then be used in inference mode to generate scores for any never-seen pair of sequences. phase 3 (optional). we fine-tune the miniature on human annotations, which, as shown in section 6, can boost performance. ## experiments in this section, frugalscore is used in inference mode to generate scores directly after pretraining, i.e., no fine-tuning is performed (see section 6 for fine-tuning results). we evaluate on two text generation tasks: summarization and translation. we use evaluation datasets containing (reference, candidate) sequence pairs annotated with human scores assessing the quality of the candidates given the references. we measure the effectiveness of frugalscore by measuring the pearson correlation of its scores with the human judgments and comparing it to that of the original metrics. we also take the number of parameters and the runtime into account. text summarization. we use 4 different multidocument summarization datasets from the text analysis conference (tac) foot_4 : tac-2008, tac-2009, tac-2010 and tac-2011. these datasets respectively contain 48, 44, 46 and 44 clusters of documents and 58, 55, 43 and 51 systems are used to generate summaries. each cluster forms a topic to be summarized and has 4 reference summaries. there are approximately 10k pairs in each dataset. each pair is annotated with two human judgment scores: the pyramid score @xcite and the responsiveness @xcite . the former measures the proportion of important semantic units (scus) in the reference summaries captured by the system summary, while the latter reflects the content coverage and the readability of each summary. machine translation. our evaluation corpus is from the wmt-2019 7 shared task @xcite . we consider all the to-english pairs: chinese, czech, german, finnish, russian, lithuanian and kazakh to english. for each language, we use the test set that contains several thousands of reference-candidate pairs annotated with human ratings that assess the translation quality. we benchmarked the metrics in terms of pearson correlations with human scores, runtimes, and numbers of parameters. we used two approaches to compute the pearson correlations: summary-level (or segment-level) and system-level. in the former approach, a score is attributed to each of the output candidates, while in the latter approach, one single overall score is attributed to the system (by averaging its individual scores). rows a to c correspond to bertscore with bert miniatures as the underlying model. they are simple baselines added for the sake of comparison, to see what we get when bertscore is used with the same number of parameters as fru-galscore. rows d to g correspond to the expensive metrics that are learned by frugalscore (in the respective sections of the bottom half of the table). they are bertscore and moverscore metrics where the underlying model is a large pretrained language model: bert-base (@xmath0, @xmath1, roberta-large (@xmath2, @xmath3, and deberta-xlarge (@xmath4, @xmath5. finally, rows i to xii correspond to fru-galscore. subscripts refer to row labels and indicate which metric-model combination was used to annotate pairs. i.e., frugalscore d learned the metric of row d, i.e., bertscore with bert-base. first, results show that all frugalscores, regardless of which metric they learned, significantly outperform the bertscores with miniature models. these results suggest that frugalscore is a better approach than using an existing metric with a lightweight underlying model. the reason for this is probably that in frugalscore, the knowledge of the original unsupervised metric (based on a large model) is explicitly transferred to the miniature via the continuation of its pretraining on the synthetic dataset. that is, the miniature is actually learning a metric. whereas, on the other hand, plugging a compressed version of a general-purpose language model into the original unsupervised metric just makes it lose expressiveness and capacity. second, we can clearly see that frugalscore retains most of the performance of the original metric, while running several times faster and reducing the number of parameters by several orders of magnitude. on average over all metrics, tasks, and miniatures, frugalscore retains 96.8% of the original performance, runs 24 times faster, and has 35 times less parameters. more precisely, on average across all metrics, frugalscore-tiny retains 97.7/94.7% of the original performance on tac (pyramid score/responsiveness), while running 54 times faster and having 84 times less parameters. its small and medium versions retain near full performance in terms of responsiveness (98 and 97.7%) and even slightly outperform the original metrics in terms of pyramid score, while at the same time reducing the runtime and the number of parameters by 32 (resp. 21) and 13 (resp. 9) times. on wmt, frugalscore-tiny retains 88.58% of the performance of the original metrics, while running 14 times faster (and still having 84 times less parameters), while the small and medium versions of frugalscore retain 95.71 % and 98.06% of the original performance while still offering a 32 times (resp. 21) speedup and having 13 times (resp. 9) less parameters, on average. interestingly, frugalscore even improves the performance of the original metrics in some cases. for example, on tac, frugalscore g with bert-tiny (row x) improves the performance of the original moverscore metric based on bert-base (row g) from 66.5 to 67.3 in terms of pyramid score, while reducing the number of parameters by 25 and running 50 times faster. other examples, also for tac with the pyramid score, include frugalscore f with bert-small (row viii, +1.5 point) and frugalscore f with bert-medium (row ix, +1 point). finally, the results of frugalscore for different miniature sizes show that, on wmt, using larger models always improves performance (e.g., row @xmath6. but interestingly, on tac, this observation does not hold (e.g., row @xmath7, and sometimes, frugalscore with the smallest miniature (bert-tiny) is superior (e.g. rows i and x). this finding suggests that the impact of the pretrained language model size is task-dependent. to sum up, results clearly show the effectiveness of frugalscore in learning a cheaper, lighter, and faster version of the original metrics, while retaining most of their original performance. the system-level correlations, provided in appendices c and d, corroborate these positive results. we also provide the correlations between the original and the learned metrics in appendices e and f. it is interesting to note that a greater correlation with the original metric is not always associated with a better performance. e.g., the tiny version of frugalscore g is the best (row x), while it is the less correlated with the original metric. ## fine-tuning on human annotations we test two hypotheses in this section: (1) whether fine-tuning on a human-annotated dataset is beneficial, and (2) when fine-tuning on human annotations, whether continuing pretraining on our synthetic dataset is useful. because we cannot use the same dataset for finetuning and evaluation, we fine-tune a bert-small on each year of tac 2008-2011 for 4 epochs, using two other years as the validation set, and the remaining year as the test set. the best epoch is selected based on validation performance. we use a batch size of 32 and a learning rate of 2e-5 that linearly decreases to zero. finally, we experiment with two scenarios: fine-tuning the miniature directly without continuing its pretraining on our synthetic dataset, and fine-tuning it after the pretraining continuation (with annotations generated by bertscore-bert-base). results. results are reported in second, even if a direct comparison is not possible, we can remark when looking at the tac pyramid score of row ii) in ## impact of data sources to test the importance of each data source introduced in subsection 3.1, we created a training set containing sequence pairs uniformly and equally sampled from each source. we annotated these pairs with the bertscore-bert-base metric and we used them to continue the pretraining of a bert-small miniature. we also considered pairs drawn at random from the pairs generated with the other strategies. the motivation for random pairs was to sample "negative examples", as seeing only "positive examples" (pairs of related sequences) could bias the learned metric towards considering any two unrelated sequences as similar. we then continued the pretraining of the bert-small miniature four times, excluding each time the pairs coming from a specific data source. we evaluated the learned metric on tac-2008 to 2011 and on wmt-2019. figure 1 shows the average improvements in the pearson correlation with human judgments relative to training a model on all sources. note that when training on all four sources, we sampled 30k pairs from each source (120k total), and when excluding a source, we sampled 40k pairs from each source (120k total). we can clearly see that excluding the random pairs improves performance while excluding any of the other data sources decreases performance. in other words, all our data sources are beneficial, and it is not necessary to add "negative examples". we hypothesise that this is due to the fact that nlg datasets typically do not contain completely unrelated pairs of sentences. interestingly, the pairs generated with the backtranslation strategy have the greatest impact on performance. ## conclusion we proposed frugalscore, an approach to learn a fixed, low-cost version of any expensive nlg evaluation metric. experiments on summarization and translation tasks show that our frugalscore versions of bertscore and moverscore retain most of the original performance in terms of the correlation with human judgments, while running several times faster and having several orders of magnitude less parameters. on average over all learned metrics, tasks, and variants, frugalscore retains 96.8% of the performance, runs 24 times faster, and has 35 times less parameters than the original metrics.