E ff E val: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics (2023)

Abstract: Efficiency is a key property to foster inclusiveness and reduce environmental costs, especially in an era of LLMs. In this work, we provide a comprehensive evaluation of efficiency for MT evaluation metrics. Our approach involves replacing computation-intensive transformers with lighter alternatives and employing linear and quadratic approximations for alignment algorithms on top of LLM representations. We evaluate six (reference-free and reference-based) metrics across three MT datasets and examine 16 lightweight transformers. In addition, we look into the training efficiency of metrics like COMET by utilizing adapters. Our results indicate that (a) TinyBERT provides the optimal balance between quality and efficiency, (b) CPU speed-ups are more substantial than those on GPU; (c) WMD approximations yield no efficiency gains while reducing quality and (d) adapters enhance training efficiency (regarding backward pass speed and memory requirements) as well as, in some cases, metric quality. These findings can help to strike a balance between evaluation speed and quality, which is essential for effective NLG systems. Furthermore, our research contributes to the ongoing efforts to optimize NLG evaluation metrics with minimal impact on performance. To our knowledge, ours is the most comprehensive analysis of different aspects of efficiency for MT metrics conducted so far.

## introduction evaluation is crucial to progress in fields such as nlp and machine learning, as it is used to identify and assess the most promising, state-of-theart approaches. it is particularly challenging for natural language generation @xcite systems as text generation is open-ended: multiple outputs, with very different surface-level realizations, can be equally correct @xcite . this insight makes classical lexical overlap metrics such as bleu @xcite or rouge @xcite figure 1 : the comet metric with pfeiffer adapter configuration. parameters of red blocks remain frozen during training, while parameters of green blocks are optimized. however, basing evaluation metrics on large transformers (and thus boosting the quality of the metrics) also has downsides: for example, the associated computational burden (i) may promote inequality among researchers, hindering diversity, as not everyone has access to expensive compute resources foot_0 and (ii) incurs high environmental costs, one of the most critical issues of our time @xcite . (iii) inefficient metrics -which are in addition non-transparent @xcite -may also prevent high-quality metrics from being deployed by the community, a potential reason why older, lower-quality but faster metrics such as bleu are still popular @xcite . to better illustrate the issue, let us consider a typical setup for evaluating machine translation (mt) systems (e.g., the setup of wmt shared tasks). in such a case, one may, for example, have 30k segments to evaluate per language pair, 5 different language pairs, and 50 assessed mt systems. if one uses the bertscore @xcite ) metric with the author-suggested roberta-large @xcite ) encoder, then it would take 71 hours to completely evaluate all the mt systems on a single nvidia a100 gpu (given that users do have access to gpus), producing around 8kg co 2 -eq of carbon footprint. if access to the gpu is restricted, one could run the metric on a cpu, but it would take more than 950 hours to do a full evaluation, producing 5.4kg co 2 -eq of carbon footprint. while in the case of a shared task/challenge, the evaluation of model outputs is usually not frequent, there are some cases where evaluation is done constantly, such as hyperparameter search and neural architecture search. apart from evaluating mt systems, evaluation metrics have a variety of possible use cases that would greatly benefit from the computationally efficient solutions: a) metrics can be used as reward functions in reinforcement learning pipelines; b) some metrics can be used in the filtering of massive, web-crawled parallel corpora; c) they can be used in an online setting for real-time re-ranking of mt systems outputs. thus, developing light-weight high-quality evaluation metrics for nlg is imperative, which we explore in depth in this work. we focus on mt as a prime instance of nlg, which also yields a diverse set of scenarios for efficiency, including training efficiency and the efficiency of multilingual models. nonetheless, we believe that our insights hold more generally. our paper about efficient evaluation (effeval) presents the following main contributions: • we provide a comprehensive analysis of inducing efficient, high-quality evaluation metrics based on three principles: (i) replacing a computationheavy transformer in the metrics by much smaller ones, obtained e.g. via pruning or distillation; ## related work our work connects to (1) transformer-based evaluation metrics and to (2) efficiency. here, we provide only a brief overview of the related papers. appendix a contains additional related work. evaluation metrics: recent transformerbased metrics utilize bert-based models like bertscore @xcite and mover-score @xcite . extensions include bartscore @xcite , which reads off probability estimates as metric scores directly from text generation systems, and menli @xcite , which uses probabilities from models fine-tuned on natural language inference task. these metrics are reference-based (comparing the mt output to a human reference), like bertscore and moverscore, or reference-free (comparing the mt output to the source text), like xmover-score @xcite and sentsim @xcite , and some are trained (fine-tuned on human scores) like comet @xcite while others are untrained, like bertscore. trained metrics typically show higher correlations with human assessments, but require more resources and, thus are more costly. transversal approaches by @xcite ; @xcite use llm predictions. efficiency: techniques like knowledge distillation @xcite , dynamic inference acceleration @xcite @xcite , and adapters @xcite seek to improve model efficiency. knowledge distillation involves a smaller student model learning from a larger teacher, e.g., distilbert @xcite and tinybert @xcite . @xcite distill an efficient evaluation metric called frugalscore from the teachers bertscore/moverscore. dynamic inference acceleration adds early exit ramps based on representation changes in encoder layers. adapters freeze pre-trained transformers and train intermediate layers, which can reduce memory usage and improve training speed with varying complexity @xcite @xcite . ## approach for optimizing the metrics, we explore three approaches: (i) we replace transformers with smaller and more efficient variants in §3.1 and (ii) we accelerate token matching by calculating word centroid distance (wcd) and relaxed word mover's distance (rwmd) instead of more computationally complex wmd for moverscore and xmoverscore in §3.2. finally, (iii) we explore the impact of using adapters on training efficiency and metric quality for comet and cometinho. ## experimental setup following related work, we measure the success of our optimized metrics based on the time needed to calculate them, the memory used, and the storage needed to save the program or related data. ## results we structure this section as follows: we report results for untrained (1) reference-based metrics ( §5.1), (2) reference-free multilingual metrics ( §5.2), (3) consider wmd approximations for untrained moverscore ( §5.3) and then (4) we investigate the impact of adapters on training efficiency and metric quality for comet and cometinho trained metrics. due to space constraints, we relegate details to the appendix c and only list the key results as a summary in the main part. ## analysis & discussion we conduct a deeper analysis on reference-based semantic-similarity metrics bertscore, mover-score and baryscore, all of which use the same efficient encoder architectures (we remove roberta- large from the analysis, as it has only been used with bertscore). we compute results individually across the three metrics and the three wmt datasets (then report individual results or averages). which speedups are obtained? to illustrate in absolute values, bertscore with roberta-large model takes, on average in a gpu environment, 34ms per segment across all three datasets. the same bertscore with tinybert takes 15ms per segment. considering the practical examples made in the introduction (30k segments × 5 language pairs × 50 mt systems), the reference bertscore would take, as we stated before, approx. 71 hours for a full pass, producing 8kg co 2 -eq of carbon footprint. our suggested alternative, tinybert, would take more than twice as little, 31 hours, producing 3.62kg co 2 -eq. in a cpu environment, the difference becomes even more striking. the reference metric with roberta-large would take more than 950 hours for a single full pass, while tinybert would complete in just under 45 hours. the carbon footprint is 5.4kg co 2eq for roberta-large and 0.26kg co 2 -eq for the tinybert-based metric (while maintaining 97% of the reference model's quality). for those calculations, we assume that a gpu environment (nvidia a100) has a power draw of 300w under full load, while a cpu environment has a power draw of 15w. however, the small size of tinybert would allow practitioners to use lower-tier gpus, including some mobile gpus in laptops, which are much more energy-efficient. for carbon footprint, we take the us 2021 power grid carbon intensity foot_3 as a reference point. how stable are the results? we correlate the stability of (normalized) results using pearson correlation over vectors corresponding to (a) metrics and (b) datasets. concerning metrics, we observe high pearson correlations from 0.72 (moverscore-baryscore, quality) to 0.99 (moverscore-bertscore, inference time). this means that the lms perform similarly (in terms of quality, inference time, or both) for each metric. across datasets, the correlation is high for wmt15-wmt16 (0.91-0.99 for quality, runtime, and average) but considerably lower between wmt21 and the other two (0.20-0.96). this is mainly because quality is not stable: for example, deebert performs badly for wmt21, but quite well for wmt15 and wmt16. we note, however, that the score ranges for wmt21 are low anyway, and deebert absolutely does not perform much lower than the other encoders here. which adapters are better? for both comet and cometinho, the 'parallel' adapter configuration consistently outperforms others, achieving high kendall τ correlation values (0.289 for comet, 0.254 for cometinho) and balancing forward and backward pass speeds. comet-inho's 'parallel' configuration also exhibits a 34% reduction in memory usage and a remarkable backward pass speed. though it remains unclear why adapter-trained models surpass standard models in terms of quality, training curves are often close and sometimes overlapping. the parallel adapter model achieves higher kendall τ values despite reduced trainable parameters, which aligns with recent findings in @xcite . our experiments utilize reference model hyperparameters, except for batch size and learning rate. thus, further hyperparameter optimization might produce even higher correlations with human assessments. ## conclusion we have investigated efficient evaluation metrics for natural language generation, particularly mt, in monolingual reference-based and multilingual reference-free versions via three approaches: (i) replacing transformers in metrics by efficient variants, (ii) replacing alignment models in metrics (precisely: word mover distance) with efficient approximations, (iii) training comet and comet-inho metrics in a parameter-efficient way with adapters. we have explored multiple types of efficient transformers, finding that tinybert shows the best quality-efficiency tradeoff for semantic similarity-based metrics: on average, it retains 97% quality while being more than 5x faster at inference time and having considerably fewer parameters and lower memory consumption. in several cases, we have also identified faster models that yield higher quality at the same time. finally, we found that efficient alignments on top of transformers do not result in efficiency gains but have adverse effects on quality in 2 out of the 3 datasets we examined. our experiments further demonstrate that the 'parallel' adapter configuration consistently outperforms others in efficiency and metric quality for both comet and cometinho. the adaptertrained models achieve faster results, using less memory, and requiring a smaller portion of the reference model's training data, with comet-sized adapter models being around 30% faster to fully train (3h30m vs. 5h4m). these findings indicate that adapter-based training offers a promising approach for natural language generation tasks, providing optimal memory usage, computational efficiency, and alignment with human assessment. in future work, we want to combine efficiency with other highly desirable properties of evaluation metrics such as robustness @xcite @xcite and explainability @xcite @xcite to induce metrics that jointly satisfy these criteria.