Searching for COMETINHO : The Little Metric That Could (2022)

Abstract: In recent years, several neural fine-tuned machine translation evaluation metrics such as COMET and BLEURT have been proposed. These metrics achieve much higher correlations with human judgments than lexical overlap metrics at the cost of computational efficiency and simplicity, limiting their applications to scenarios in which one has to score thousands of translation hypothesis (e.g. scoring multiple systems or Minimum Bayes Risk decoding). In this paper, we explore optimization techniques, pruning, and knowledge distillation to create more compact and faster COMET versions. Our results show that just by optimizing the code through the use of caching and length batching we can reduce inference time between 39% and 65% when scoring multiple systems. Also, we show that pruning COMET can lead to a 21% model reduction without affecting the model’s accuracy beyond 0.01 Kendall tau correlation. Furthermore, we present DISTIL-COMET a lightweight distilled version that is 80% smaller and 2.128x faster while attaining a performance close to the original model and above strong baselines such as BERTSCORE and PRISM.

## introduction traditional metrics for machine translation (mt) evaluation rely on lexical similarity between a given hypothesis and a reference translation. metrics such as bleu @xcite and chrf @xcite remain popular due to efficient memory usage and fast computational performance, even though several studies have shown that they correlate poorly with human judgements, specially for high quality mt @xcite . in contrast, neural fine-tuned metrics on top of pre-trained models such as mbert @xcite and xlm-r @xcite ) (e.g bleurt @xcite and comet @xcite have demonstrated significant improvements in comparison to other metrics @xcite @xcite . the improvements made them good candidates for revisiting promising research directions where the metric plays a more central role in candidate selection during decoding, such as n -best reranking @xcite @xcite and minimum bayes risk (mbr) decoding @xcite . nonetheless, the complexity of such strategies using metrics based on large transformer models can become impractical for a large set of mt hypotheses. in this paper, we describe several experiments that attempt to reduce comet computational cost and model size to make it more efficient at inference. our techniques are particularly useful in settings where we have multiple translations from different systems on the same source sentences. since the models are based on triplet encoders, we will first analyse the impact of embedding caching and length batching. then, we will try to further reduce the computational cost by using weight pruning and knowledge distillation. our results show that embedding caching and length batching alone can boost comet performance 39.19% when scoring one system and 65.44% when scoring 8 systems over the same test set. furthermore, with knowledge distillation we are able to create a model that is 80% smaller and 2.128x faster with a performance close to the original model and above strong baselines such as bertscore and prism. figure 1 shows time differences for all proposed methods when evaluating a varying number of systems. ## related work in the last couple of years, learned metrics such as comet @xcite and bleurt @xcite proved to achieve high correlations with human judgments @xcite @xcite . they are cast as a regression problem and capture the semantic similarity between the translated text and a reference text, going beyond the simple surface/lexical similarities-the base of popular metrics like bleu @xcite and chrf @xcite . the fact that comet and bleurt metrics leverage large pre-trained multilingual models was a huge turning point. by using contextual embeddings trained on a different task, researchers were able to overcome the scarcity of data in mt evaluation (as well as in other tasks in which data is also limited). with such multilingual models, high-quality mt evaluation is now a possibility, even for language pairs without labeled data available (i.e. zero-shot scenarios). however, this multilingual property usually comes with a trade-off. for example, for cross-lingual transfer task, gains in performance (higher accuracy with human labels) only occur by adding new language pairs until a certain point, after which adding more languages actually decreases the performance, unless the model capacity is also increased (a phenomena called "the curse of multilinguality" @xcite . besides the curse of multilinguality phenomena, the nlp community has been motivated to build larger and larger transformer models because, generally, the bigger the model the better it performs. this was demonstrated in several tasks like the ones in the glue benchmark @xcite and in multilingual translation tasks @xcite . hence, models are achieving astonishing sizes like bert with 340m parameters @xcite , xlm-r xxl with 10.7b parameters @xcite , m2m-100 with 12b parameters @xcite , and gpt-3 with 175b parameters @xcite . however, this growth comes with computational, monetary and environmental costs. for example, training a model with 1.5b parameters costs from 80k dollars up to 1.6m dollars foot_0 when doing hyper-parameter tuning and performing multiple runs per setting @xcite . such scale makes running similar experiments impractical to the majority of research groups, and the high energy and high response latency of such models are preventing them from being deployed in production (e.g. @xcite ). to deal with the above problem, it is necessary to apply techniques for making models more compact, such as pruning, distillation, quantization, among others. in a recent review @xcite summarizes these techniques for increasing inference efficiency, i.e., for making the model faster, consuming fewer computational resources, using less memory, and less disk space. distilbert @xcite is a successful example: using distillation with bert as the teacher and reducing the amount of layers from the regular 12 to only 6, the model retains 97% of bert's performance while reducing the size by 40% and being 60% faster. the authors have also shown that when used for a mobile application (iphone), the distilbert was 71% faster than bert. another example, closer to our research, is the metric obtained from using synthetic data and performing distillation using a new variation of bleurt as the teacher @xcite . the resulting metric obtains up to 10.5% improvement over vanilla fine-tuning and reaches 92.6% of teacher's performance using only a third of its parameters. nonetheless, the architecture of bleurt-based models requires that the reference is always encoded together with mt hypothesis which is extremely inefficient in use cases such as mbr, where the metric has a o(n foot_2 ) complexity (with n being the number of hypotheses), and system scoring where for a fixed source and reference we can have several translations being compared. ## length sorting and caching before exploring approaches that reduce the number of model parameters, we experiment with techniques to optimize the inference time computational load. one which is commonly used is to sort the batches according to sentence length to reduce tensor padding @xcite . since comet receives three input texts (source, hypothesis and reference), for simplicity, we do length sorting according to the source length. figure 2 shows the speed difference between an unsorted test set with varying size and length-based sorting. as previously pointed out, comet metrics are based on triplet encoders 2 which means that the source and reference encoding does not depend on the provided mt hypothesis as opposed to other recent metrics such as bleurt @xcite which have to repetitively encode the reference for every hypotheses. with that said, using comet we only need to encode each unique sentence (source, hypothesis translation or reference translation) once. this means that we can cache previously encoded batches and reuse their representations. in figure 3 , we show the speed gains, in seconds, when scoring multiple systems over the same test set. this reflects the typical mt development use case in which we want to select the best among several mt systems. these two optimizations altogether are responsible for reducing the inference time of comet from 34.7 seconds to 21.1 seconds while scoring 1 system (39.19% faster) and from 265.9 seconds to 91.9 seconds when scoring 8 systems (65.44% faster). for all experiments performed along the rest of the paper we always use both optimization on all comet models being compared. ## model pruning model pruning has been widely used in natural language processing to remove non-informative connections and thus reducing model size @xcite . since most comet parameters come from the xlm-r model, we attempt to reduce its size. we start with layer pruning by removing the top layers of xlm-r. then we experiment with making its encoder blocks smaller either by reducing the size of the feed-forward hidden layers or by removing attention heads. the main advantage of these approaches is their simplicity: within minutes we are able to obtain a new model with reduced size and memory footprint with minimal performance impact. for all the experiments in this section, we used the development set from the metrics shared task of wmt 2020. this set contains direct assessment annotations (da; @xcite ) for @xmath0, @xmath1, @xmath2. we use these language pairs because they were annotated by experts exploring document context and in a bilingual setup (without access to a reference translation) foot_3 . nonetheless, in section 6 we show the resulting model performance on all language pairs from wmt 2021 for both da and multidimensional quality metric annotations (mqm; @xcite ). ## distillation another commonly used way to compress neural networks is through knowledge distilation @xcite in which, for large amounts of unlabeled data, a smaller neural network (the student) is trained to mimic a more complex model (the teacher). as the teacher network, we used an ensemble of 5 comet models trained with different seeds @xcite . the student network follows the same architecture as the original model and the same hyper-parameters. however, instead of using xlm-r-large, it uses a distilled version with only 12 layers, 12 heads, embeddings of 384 features, and intermediate hidden sizes of 1536. this model has only 117m parameters compared to the 560m parameters from the large model. regarding the unlabeled data for distillation, we extracted 25m sentence pairs from opus (tiedemann, 2012) ranging a total of 15 language pairs. to guarantee high quality parallel data we used bicleaner tool @xcite with a threshold of 0.8. then, using pre-trained mt models available in hugging face transformers, we created 2 different translations for each source: one using a bilingual model (in theory a high quality translation) and another using pivoting (which can be thought as lower quality). finally, we scored all the data using our teacher ensem- ## correlation with human judgements in this section, we show results for {prune and distil}-comet in terms of correlations with mqm annotations from wmt 2021 metrics task for two different domains: news and ted talks. since these annotations only cover high-resource language pairs (@xmath3, @xmath4, @xmath5, we also evaluate models on low resource language pairs using da relative ranks from wmt 2021, namely we test these models for: hindi↔bengali, zulu↔xhosa, @xmath6, @xmath7, @xmath8. for a detailed comparison, we also present results for chrf @xcite and bleu @xcite , two computationally efficient lexical metrics, and other neural met-rics such as prism foot_6 @xcite , bleurt @xcite and bertscore @xcite . from in minimum bayes risk (mbr) decoding, a machine translation evaluation metric can be used as the utility function for comparing the translation hypotheses. this kind of approach, also known as "consensus decoding", derived from the idea that the top ranked translation is the one with the highest average score when compared to all other hypotheses. this process requires that each hypothesis translation be compared to every other hypotheses in an hypotheses candidate list. having faster neural metrics could directly impact research and computational performance of using mbr decoding approaches with such metrics. using comet models with distillation or pruning can have a considerable effect at the performance of mbr decoding using such models as the utility function. figure 7 shows that distil-comet is always substantially faster than the original comet model especially for larger candidate list sizes such as 200 candidates. likewise, prune-comet performs better than the original model but its performance is also considerably higher than distil-comet. regarding the two comet variants there is a clear trade-off that needs to be taken into consideration, as evidenced by the results in section 6: while distil-comet is faster, prune-comet is 7 for a more detailed comparison between comet and bleurt metrics we refer the reader to the wmt 2021 metrics shared task results paper @xcite where both metrics ended up statistically tied for most language pairs and domains. more accurate, leaving the choice of each model to use up to the most important aspect for the application. in the case of mbr decoding, this might depend on the hardware available for performing the computations. ## conclusion and future work in this paper we presented two simple optimizations that lead to significant performance gains on neural metrics such as comet and two approaches to reduce its number of parameters. together these techniques achieve impressive gains in performance (both speed and memory) at a very small cost in performance. to showcase the effectiveness of our methods, we presented distil-comet and prune-comet. these models were obtained using comet knowledge distillation and pruning respectively. to test the proposed models, we used the data from the wmt 2021 metrics task which covers low resource languages as well as high resource languages. overall the results of prune-comet are stable across the board with only a small degradation compared to the original metric. knowledge distillation leads to much higher compression rates but seems to confirm previous findings @xcite which suggest the lack of model capacity when it comes to the multilingual generalization for low resource languages. a primary avenue for future work is to study how decreasing the model size can further impact on robustness of the metric, inspired by recent studies which identified weaknesses of comet metrics when dealing with numbers and named entities @xcite . also, in this work we explored knowledge distillation directly from the teacher output but an interesting avenue for improving the quality of the student model is to explore alternative distillation approaches that learn directly from internal representations of the teacher model such as self-attention distillation @xcite .