Learning Compact Metrics for MT (2021)

Abstract: Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on related tasks suggest that these models are most efficient when they are large, which is costly and impractical for evaluation. We investigate the trade-off between multilinguality and model capacity with RemBERT, a state-of-the-art multilingual language model, using data from the WMT Metrics Shared Task. We present a series of experiments which show that model size is indeed a bottleneck for cross-lingual transfer, then demonstrate how distillation can help addressing this bottleneck, by leveraging synthetic data generation and transferring knowledge from one teacher to multiple students trained on related languages. Our method yields up to 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT’s performance using only a third of its parameters.

## introduction recent improvements in machine translation (mt) and multilingual natural language generation (nlg) have led researchers to question the use of n-gram overlap metrics such as bleu and rouge @xcite . since these metrics focus solely on surface-level aspects of the generated text, they correlate poorly with human evaluation, especially when models are producing high-quality text @xcite @xcite . this has led to a surge of interest in learned metrics that cast evaluation as a regression problem and leverage pre-trained multilin-gual models to capture the semantic similarity between references and generated text @xcite . popular examples of those metrics include comet @xcite and bleurt-extended @xcite , based on xlm-roberta @xcite and mbert @xcite respectively. these metrics deliver superior performance over those based on lexical overlap, outperforming even crowd-sourced annotations @xcite . large pre-trained models benefit learned metrics in at least two ways. first, they allow for cross-task transfer: the contextual embeddings they produce allow researchers to address the relative scarcity of training data that exist for the task, especially with large models such as bert or xlnet @xcite @xcite . second, they allow for cross-lingual transfer: mt evaluation is often multilingual, yet few, if any, popular datasets cover more than 20 languages. evidence suggests that training on many languages improves performance on languages for which there is little training data, including the zero-shot setup, in which no fine-tuning data is available @xcite @xcite . however, the accuracy gains only appear if the model is large enough. in the case of cross-lingual transfer, this phenomenon is known as the curse of multilinguality: to allow for positive transfer, the model must be scaled up with the number of languages @xcite . scaling up metric models is particularly problematic, since they must often run alongside an already large mt or nlg model and, therefore, must share hardware resources (see @xcite for a recent use case). this contention may lead to impractical delays, it increases the cost of running experiments, and it prevents researchers with limited resources from engaging in shared tasks. we first present a series of experiments that validate that previous findings on cross-lingual transfer and the curse of multilinguality apply to the metrics domain, using rembert (rebalanced mbert), a multilingual extension of @xcite . we then investigate how a combination of multilingual data generation and distillation can help us reap the benefits of multiple languages while keeping the models compact. distillation has been shown to successfully transfer knowledge from large models to smaller ones @xcite , but it requires access to a large corpus of unlabelled data @xcite , which does not exist for our task. inspired by @xcite , we introduce a data generation method based on random perturbations that allows us to synthesize arbitrary amounts of multilingual training data. we generate an 80m-sentence distillation corpus in 13 languages from wikipedia, and show that we can improve a vanilla pre-trained distillation setup @xcite by up to 12%. a second, less explored benefit of distillation is that it lets us partially bypass the curse of multilinguality. once the teacher (i.e., larger) model has been trained, we can generate training data for any language, including the zero-shot ones. thus, we are less reliant on cross-lingual transfer. we can lift the restriction that one model must carry all the languages, and train smaller models, targeted towards specific language families. doing so increases performance further by up to 4%. combining these two methods, we match 92.6% of the the largest rembert model's performance using only a third of its parameters. a selection of code and models is available online at https://github.com/ google-research/bleurt. ## multilinguality and model size to motivate our work, we quantify the trade-off between multilinguality and model capacity using data from the wmt shared task 2020, the most recent benchmark for mt evaluation metrics. the phenomenon has been well-studied for tasks such as translation @xcite and language inference @xcite , but it is less well understood in the context of evaluation metrics. task and data in the wmt metrics task, participants evaluate the quality of mt systems with automatic metrics for 18 language pairs-10 to-english, 8 from-english. the success cri- figure 1 : performance of our models. the dashed line represents the performance of @xcite . the metric is wmt metrics darr @xcite , a robust variant of kendall tau, higher is better. we run each experiment with 5 random seeds, report the mean result and normal-based 95% confidence intervals. models like comet @xcite and bleurt @xcite , we treat evaluation as a regression problem where, given a reference translation x (typically produced by a human) and predicted translation x (produced by an mt system), the goal is to predict a real-valued human rating y. as is typical, we leverage pretrained representations @xcite to achieve strong performance. specifically, we first embed sentence pairs into a fixed-width vector @xmath0, x) using a pretrained model f and use this vector as input to a linear layer: ŷ = wv + b, where w and b are the weight matrix and bias vector respectively. for the pretrained model f, we use rem-bert @xcite , a recently published extension of mbert @xcite pretrained on 104 languages using a combination of wikipedia and mc4 @xcite . be- figure 2 : impact of the number of fine-tuning languages on zero-shot performance, using rembert-6 and rembert-32 on en-ja, en-pl, and en-ta. cross-lingual transfer during fine-tuning figure 2 displays the performance of rembert-6 and rembert-32 on the zero-shot languages as we increase the number of languages used for finetuning. we start with english, then add the languages cumulatively, in decreasing order of frequency (without adding data for any of the target languages). cross-lingual transfer works: in all cases, adding languages improves performance. the effect is milder on rembert-6, which consistently starts higher but finishes lower. the appendix presents additional details and results. capacity bottleneck in pre-training to further understand the effect of multilinguality, we pretrained the smaller models from scratch using 18 languages of wmt instead of 104, and fine-tuned on the whole dataset. figure 3 presents the results: performance increases in all cases, especially for teacher pre-training model: rembert-32 data: mc4, wiki. task: lang. model teacher fine-tuning model: rembert-32 data: wmt metrics task: mt eval teacher inference model: rembert-32 data: unlabelled distillation corpus student pre-training models: rembert-3,6,12 data: wikipedia task: lang. model student distillation models: rembert-3,6,12 data: labelled distillation corpus rembert-3. this suggests that the models are at capacity and that the 100+ languages of the pretraining corpus compete with one another. takeaways learned metrics are subject to conflicting requirements. on one hand, the opportunities offered by pre-training and cross-lingual transfer encourage researchers to use large, multilingual models. on the other hand, the limited hardware resources inherent to evaluation call for smaller models, which cannot easily keep up with massively multilingual pre-training. we address this conflict with distillation. ## addressing the capacity bottleneck since pre-trained distillation was shown to be simple and efficient, we use it for our base setup. figure 4 summarizes the steps: we fine-tune rembert-32 on human ratings, run it on an unlabelled distillation corpus, and use the predictions to supervise rembert-3, 6, or 12. by default, we use the wmt corpus for distillation, i.e., we use the same sentence pairs for teacher fine-tuning and student distillation (but with different labels). improvement 1: data generation distillation requires access to a large multilingual dataset of sentence pairs (reference, mt output) to be annotated by the teacher. yet the wmt metrics corpus is relatively small, and no larger corpus exists in the public domain. to address this challenge we generate pseudo-translations by perturbing sentences from wikipedia. we experiment with three types of perturbations: back-translation, word substitutions with mbert, and random deletions. the motivation is to generate surface-level noise and paraphrases, to expose the student to the different types of perturbations that an mt system could introduce. in total, we generate 80 million sentence pairs in 13 languages. the approach is similar to @xcite , who use perturbations to generate pre-training data in english. we present the details of the approach in the appendix. improvement 2: 1-to-n distillation another benefit of distillation is that it allows us to lift the constraint that one model must carry all the languages. in a regular fine-tuning setup, it is necessary to pack as many languages as possible in the same model because training data is sparse or non-existent in most languages. in our distillation setup, we can generate vast amounts of data for any language of wikipedia. it is thus possible to bypass the capacity constraint by training n specialized students, focused on a smaller number of languages. for our experiments, we pre-train five versions of each rembert, which cover between 3 and 18 languages each. we tried to form clusters of languages that are geographically close or linguistically related (e.g., germanic or romance languages), such that each cluster would cover at least one language of wmt. we list all the languages in the appendix. results on en-*, the improvements are cumulative: distill wmt+wiki outperforms distill wmt (between 5 and 12% improvement), and it is itself outperformed by 1-to-n (up to 4%). combining techniques improves the baselines in all cases, up to 10.5% improvement compared to fine-tuning. rembert-12 matches 92.6% of the teacher model's performance using only a third of its parameters, and it is competitive with current state-of-the-art models. runtime to validate the usefulness of our approach, we illustrate how to speed up rembert in figure 5 . we obtain a first 1.5-2x speedup compared to rembert-32 by applying length-based batching, a simple optimization which consists in batching examples that have similar a length and cropping the resulting tensor, as done in bert-score @xcite . doing so allows us to remove the padding tokens, which cause wasteful computations. we obtain a further 1.5x speedup by using the distilled version of the model, rembert-12. the final model processes 4.8 tuples per second without gpu (86 with a gpu), an 2.5-3x improvement over rembert-32. note that rembert-32 and comet are both based on the transformer architecture (we used a comet checkpoint based on xlm-r large), and rembert-32 is larger than comet. we hypoth-esize that the performance gap comes from differences in implementation and model architecture; in particular, rembert-32 has an input sequence length of 128 while xlm-r operates on sequences with length 512. ## conclusion we experimented with cross-lingual transfer in learned metrics, exposed the trade-off between multilinguality and model capacity, and addressed the problem with distillation on synthetic data. further work includes generalizing the approach other tasks and experimenting with complementary compression methods such as pruning and quantization @xcite , as well as increasing linguistic coverage @xcite .