x COMET -lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics (2024)

Abstract: State-of-the-art trainable machine translation evaluation metrics like xCOMET achieve high correlation with human judgment but rely on large encoders (up to 10.7B parameters), making them computationally expensive and inaccessible to researchers with limited resources. To address this issue, we investigate whether the knowledge stored in these large encoders can be compressed while maintaining quality. We employ distillation, quantization, and pruning techniques to create efficient xCOMET alternatives and introduce a novel data collection pipeline for efficient black-box distillation. Our experiments show that, using quantization, xCOMET can be compressed up to three times with no quality degradation. Additionally, through distillation, we create an 278M-sized xCOMET-lite metric, which has only 2.6% of xCOMET-XXL parameters, but retains 92.1% of its quality. Besides, it surpasses strong small-scale metrics like COMET-22 and BLEURT-20 on the WMT22 metrics challenge dataset by 6.4%, despite using 50% fewer parameters. All code, dataset, and models are available online.

## introduction automatic evaluation metrics are crucial for reliably measuring the quality of responses from natural language generation (nlg) systems. researchers and practitioners working on tasks such as machine translation (mt), summarization, poetry generation, etc., routinely use metrics to assess their systems' quality. apart from directly assessing the systems, evaluation metrics have many other applications: a) filtering web-scale datasets @xcite ; b) using metrics as reward functions for reinforcement learning @xcite ; c) online re-ranking of outputs of multiple systems to choose the best response to return to the user @xcite . with generative models' growing sizes and complexity, automatic evaluation metrics also evolve figure 1 : xcomet can be distilled into a small model, which will be 6-7 percentage points better than sota models with comparable parameter count. and become more computationally expensive. in the last few years, for mt evaluation, researchers have moved from traditional n-gram and characterbased metrics, such as bleu @xcite and chrf @xcite , to embeddingbased metrics, such as bertscore @xcite and moverscore @xcite , to learned metrics, which provide state-of-the-art correlation with human judgment. according to @xcite , the best-performing metrics for mt evaluation are xcomet @xcite , metricx @xcite , and gemba-mqm @xcite . all those metric models have a large number of parameters: xcomet and metricx have 10.7b-13b parameters, while gemba-mqm relies on the large language model (llm) gpt4 (openai, 2023), for which the number of parameters is unknown but speculated to be around 1.7t foot_0 . the lack of efficient alternatives to these models creates a disparity in access among researchers. under-resourced labs, students, startups, and hobbyists without access to top-tier accelerators (with more than 22gb vram) or financial resources for paid apis cannot employ those metrics. those with access to such resources may also experience prolonged iteration time due to the computation needed for those models. this is especially noticeable in the case of repeated evaluations during the hyperparameter optimization or processing of large-scale datasets. for instance, running the xcomet-xxl model to filter a crawled dataset of 10 7 examples would take 142.2 hours on a capable consumer-grade gpu, requiring 42.6 kwh of electricity and emitting around 15.6 kg co 2 -eq. 2 thus, developing alternative efficient metrics is now more vital than ever. in this paper, we explore various techniques to develop an efficient alternative to the state-of-theart xcomet metric for evaluating mt quality. our approach focuses on three main methods: knowledge distillation, quantization, and pruning. knowledge distillation is a method of creating capable small deep fitted models by training them on the outputs of the larger model. we apply knowledge distillation @xcite , training a smaller version of the xcomet model on large amounts of data, using labels created by the original xcomet-xxl model. quantization reduces the precision of deep learning model parameters and activations from 32/16 bits into 8, 4, 3, and 2 bits, occupying less memory and allowing for faster computations. pruning involves the removal of less significant parts of the model, either specific parameters, blocks of parameters, or entire layers. we apply layer pruning together with subsequent fine-tuning, which allows for accelerated inference throughput and helps mitigate potential accuracy loss. by exploring distillation, quantization, and pruning, as well as their combinations, we aim to create an efficient alternative to xcomet that maintains a high level of quality while substantially reducing hardware requirements. our main contributions are as follows: a) we conduct a comprehensive study of different compression methods (knowledge distillation, quantization, and pruning) and their interactions for the stateof-the-art mt evaluation metric xcomet. to the best of our knowledge, this is the first work to systematically investigate the effectiveness and tradeoffs of these techniques when applied to a largescale, complex metric like xcomet; b) we introduce a novel data collection pipeline for preparing large-scale, high-quality datasets for black-box distillation of xcomet. we collect 14m examples with translation hypotheses of varying qual-ity paired with high-quality reference translations. this enables the distilled model to effectively transfer the evaluation capabilities of the teacher model, xcomet-xxl; c) through our distillation method, we develop xcomet-lite, a lightweight yet highly effective mt evaluation metric. xcomet-lite achieves state-of-the-art quality among metrics @xmath0, surpassing the previous best model, comet-22, while being substantially smaller; d) we explore the use of quantization for compressing xcomet and demonstrate that 3-bit quantization can effectively reduce hardware requirements for 3b and 11b model versions without compromising quality; e) we investigate the effectiveness of pruning for compressing xcomet and show that while pruning up to 25% of the model layers can improve inference speed and memory consumption with only a marginal impact on quality, removing more layers leads to substantial quality degradation. f) we conduct a novel study of the interactions between compression methods, revealing that distillation combines well with quantization but is incompatible with pruning in our experiments. ## related work recent work has explored improving the transparency and capabilities of mt evaluation metrics. @xcite introduced metricx. this learned regression-based metric achieves state-ofthe-art correlations with human judgments through multi-stage fine-tuning on direct assessment data, consolidated mqm scores, and small-scale synthetic corpora, which is used to boost robustness. it is based on the mt5-xxl encoder-decoder model with 11b parameters. @xcite proposed gemba-mqm, which leverages the gpt-4 language model with a few-shot prompting approach to identify translation error spans and categories. this enables detailed error analysis, though reliance on the computationally expensive proprietary gpt-4 llm poses challenges for academic research. @xcite developed xcomet, a learned metric based on the xlm-roberta-xl/xxl encoder that bridges sentencelevel quality prediction with fine-grained error span detection. by training on direct assessment and mqm data, xcomet achieves top quality on sentence-level, system-level, and error span prediction tasks while providing interpretability through its predicted error spans. previously, researchers have also explored techniques for creating more efficient mt evaluation metrics while preserving their correlation with human judgments. kamal eddine et al. (2022) proposed frugalscore, which learns lightweight versions of metrics like bertscore and moverscore using knowledge distillation. their distilled metrics perform similarly to the originals while being much faster and having orders of magnitude fewer parameters. @xcite introduced cometinho, a more compact and faster version of the comet metric. they optimize the comet code using caching and length batching and further compress the model using pruning and knowledge distillation on synthetic data. the resulting model is 80% smaller and over 2 times faster than the original while maintaining competitive quality. ## methods we explore three compression techniques to develop an efficient alternative to xcomet for evaluating mt quality: quantization, pruning, and distillation. these methods aim to reduce the computational requirements and improve the inference speed of xcomet while maintaining a high level of quality. quantization quantization is a highly effective compression method with two main approaches: quantization-aware training (qat) and post-training quantization (ptq) @xcite . qat offers better prediction quality but requires costly training, making ptq more popular. ptq is further divided into data-free and data-aware methods, where the latter relies on calibration to estimate the data distribution parameters for higher prediction quality. another distinction is weight-only quantization and weight & activation quantization, with the second approach having slightly lower prediction quality but potential for faster computations using efficient 8-or 4-bit cuda kernels. in a nutshell, the quantization process comes down to finding bias and scale for each floating point value @xmath1, β] to convert it to a n-bit integer x @xmath2, β q ]: dynamic quantization @xcite is a technique that generates the zero-point x 0 and scale σ parameters in real-time, thereby eliminating the need for calibration data. due to the unknown distribution parameters, activations are maintained in floating-point format. the process of obtaining quantization parameters (α, β) and quantizing floating-point tensors to integer tensors is relatively straightforward, with the necessary statistics being computed during inference. among data-free quantization methods, llm.int8() @xcite and qlora @xcite stand out as the most prominent. (i) llm.int8() quantizes model weights to 8-bit precision using the absmax quantization technique. this method also dynamically quantizes activations to enable efficient matrix multiplications primarily in int8, with certain calculations performed in fp16 for precision. (ii) qlora uses a more advanced double quantization approach. it utilizes the nf4 data type for storage, minimizing memory demands, while computation is conducted in higher precision types (fp16, bf16), dequantizing weights on a per-layer basis. gptq @xcite is an example of weight-only quantization methods. it performs layer-by-layer quantization, minimizing the squared error relative to the full precision layer output: here, w are the full precision weights, x denotes the layer input corresponding to a small set of m data points running through the network, w represents a matrix of quantized weights, and ∥ • ∥ f is the frobenius norm. pruning pruning is the removal of the least significant parts of the neural network. it can be divided into structured and unstructured. the latter proves helpful on a cpu but is rarely practical on a gpu, since gpus are heavily optimized for dense matrix multiplication. structured pruning can take many forms, from enforcing 2:4 sparsity patterns (in each contiguous block of four values, two values must be zero) to pruning channels or entire blocks of the networks. inspired by recent works on layer pruning in llms @xcite which remove 25-50% of layers with moderate quality drop, we test its applicability for inducing efficient metrics. specifically, we adopt a simple pruning technique, described in sec. 4.4 of @xcite : in an l-layer model, we drop layers (ln) to (l -1). this heuristic is based on the observations that pruning deeper layers should affect the model less, as fewer layers rely on changes made by this layer, but also that the ultimate layer is especially important as it "decodes" the hidden states for the last fragment of the network, and cannot be removed. to mitigate the quality drop incurred by layer removal, we apply parameterefficient fine-tuning. concretely, we fine-tune all biases in linear layers, layernorm affine parameters, layerwise attention weights, and the regression and tagging heads of xcomet. this is akin to the bitfit @xcite sparse-fine-tuning approach, and has the benefit of adding no parameters and being extremely simple to implement. we also evaluate magnitude pruning and wanda pruning @xcite . in magnitude pruning, the importance of each weight s ij is directly estimated by its magnitude |w ij |. wanda pruning refines this approach by weighting each |w ij | by the average l2 norm of its corresponding input features, 1 n n @xmath3, aiming to provide a more informed measure of importance. in both methods, the weights with the lowest importance scores are pruned according to the specified sparsity pattern (unstructured, 2:4 or 4:8). ## experiments we compare quantization, pruning, and distillation for compressing xcomet. we compare it to both released versions, -xl and -xxl. as we focus on computational efficiency, we measure the model (i) inference speed, (ii) resource requirements (in terms of gpu memory, vram), and (iii) metric prediction quality, expressed in kendall-τ correlation with human judgment. ## discussion the compression methods applied to xcomet-xl and xcomet-xxl models demonstrate the potential for reducing memory consumption and increasing processing speed while maintaining competitive prediction quality. quantization methods, particularly gptq 8-bit and 3-bit, achieve substantial memory savings without compromising the models quality. quantization can also be combined with distillation with little-to-no quality reduction. pruning methods, while capable of reducing memory consumption and increasing throughput, result in a more noticeable decrease in correlation compared to quantization. our results align with the findings in @xcite , which conclude that up to 5 out of 24 layers of encoder model can be removed without noticeable quality degradation of the metric. at the same time, the layer pruning works slightly worse than in other tasks @xcite , where up to 50% of layers could be removed for large models. pruning appears incompatible with our distilled model, due to a substantial drop in metric quality. magnitude pruning with 4:8 sparsity pattern shows promising results with respect to quality / speedup trade-off. moreover, it potentially offers almost 50% reduction in peak memory consumption (and e.g. torch library will likely support structured spasity formats quite soon). the distillation of xcomet-xxl into the smaller mdeberta-based model, xcomet-lite, is a highly effective approach for improving computational efficiency while maintaining competitive metric quality. our distillation method, based on collecting large-scale diverse dataset, proves successful for distilling the xcomet metric and is easily scalable to additional translation directions. when considering speed, the distilled xcometlite outperforms other compression methods, processing a substantially higher number of samples per second on both consumer-grade rtx 3090 and hpc-grade a100 gpus. pruning is the next best performer, allowing for up to 1.3-1.5 times speedup while maintaining competitive metric quality. ## conclusion in the rapidly evolving field of mt evaluation, the current top-performing metrics, such as metricx, xcomet, and gemba-mqm, are all based on extremely large underlying models. these models, including mt5 with 13b parameters, xlm-roberta-xxl with 11b parameters, and the closed-source gpt-4 with an estimated 1.7t parameters, pushed the boundaries of performance but come with substantial computational costs and hardware requirements. our research aims to address these challenges by comparing three commonly used compression methods -quantization, pruning, and knowledge distillation -in compressing the xcomet model. we have demonstrated that these methods can effectively reduce memory consumption and increase processing speed while maintaining competitive performance, making them viable options for deploying large state-of-the-art learned metric for mt evaluation in a resource-constrained environments. in particular, our distilled model xcometlite achieves competitive prediction quality with a substantially smaller model size, offering a solution for researchers and practitioners with no access to top-tier hardware. based on our findings, we recommend the following: for the highest quality with a reduced vram requirements, opt for 8-bit or 3-bit quantization with gptq. for improved speed without substantial quality penalty, test 4-bit quantization with qlora, try structured magnitude pruning (2:4, 4:8) or prune up to 25% of the model layers.for massive speedup and low hardware requirements, consider the distilled model xcomet-lite or its quantized version, accepting a slight compromise on quality. the choice of compression method ultimately depends on the hardware, amount of data, and acceptable quality loss.