BLEURT : Learning Robust Metrics for Text Generation (2020)

Abstract: Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgment. We propose BLEURT, a learned evaluation metric for English based on BERT. BLEURT can model human judgment with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG data set. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.

## introduction in the last few years, research in natural text generation (nlg) has made significant progress, driven largely by the neural encoder-decoder paradigm @xcite which can tackle a wide array of tasks including translation @xcite , summarization @xcite , structureddata-to-text generation @xcite @xcite dialog @xcite and image captioning @xcite . however, progress is increasingly impeded by the shortcomings of existing metrics @xcite @xcite . human evaluation is often the best indicator of the quality of a system. however, designing crowd sourcing experiments is an expensive and high-latency process, which does not easily fit in a daily model development pipeline. therefore, nlg researchers commonly use automatic evaluation metrics, which provide an acceptable proxy for quality and are very cheap to compute. this paper investigates sentence-level, referencebased metrics, which describe the extent to which a candidate sentence is similar to a reference one. the exact definition of similarity may range from string overlap to logical entailment. the first generation of metrics relied on handcrafted rules that measure the surface similarity between the sentences. to illustrate, bleu @xcite and rouge @xcite , two popular metrics, rely on n-gram overlap. because those metrics are only sensitive to lexical variation, they cannot appropriately reward semantic or syntactic variations of a given reference. thus, they have been repeatedly shown to correlate poorly with human judgment, in particular when all the systems to compare have a similar level of accuracy @xcite @xcite . increasingly, nlg researchers have addressed those problems by injecting learned components in their metrics. to illustrate, consider the wmt metrics shared task, an annual benchmark in which translation metrics are compared on their ability to imitate human assessments. the last two years of the competition were largely dominated by neural net-based approaches, ruse, yisi and esim @xcite . current approaches largely fall into two categories. fully learned metrics, such as beer, ruse, and esim are trained end-to-end, and they typically rely on handcrafted features and/or learned embeddings. conversely, hybrid metrics, such as yisi and bertscore combine trained elements, e.g., contextual embeddings, with handwritten logic, e.g., as token alignment rules. the first category typically offers great expressivity: if a training set of human ratings data is available, the metrics may take full advantage of it and fit the ratings distribution tightly. fur-thermore, learned metrics can be tuned to measure task-specific properties, such as fluency, faithfulness, grammar, or style. on the other hand, hybrid metrics offer robustness. they may provide better results when there is little to no training data, and they do not rely on the assumption that training and test data are identically distributed. and indeed, the iid assumption is particularly problematic in nlg evaluation because of domain drifts, that have been the main target of the metrics literature, but also because of quality drifts: nlg systems tend to get better over time, and therefore a model trained on ratings data from 2015 may fail to distinguish top performing systems in 2019, especially for newer research tasks. an ideal learned metric would be able to both take full advantage of available ratings data for training, and be robust to distribution drifts, i.e., it should be able to extrapolate. our insight is that it is possible to combine expressivity and robustness by pre-training a fully learned metric on large amounts of synthetic data, before fine-tuning it on human ratings. to this end, we introduce bleurt, 1 a text generation metric based on bert @xcite . a key ingredient of bleurt is a novel pre-training scheme, which uses random perturbations of wikipedia sentences augmented with a diverse set of lexical and semantic-level supervision signals. to demonstrate our approach, we train bleurt for english and evaluate it under different generalization regimes. we first verify that it provides state-of-the-art results on all recent years of the wmt metrics shared task (2017 to 2019, to-english language pairs). we then stress-test its ability to cope with quality drifts with a synthetic benchmark based on wmt 2017. finally, we show that it can easily adapt to a different domain with three tasks from a data-to-text dataset, webnlg 2017 @xcite . ablations show that our synthetic pretraining scheme increases performance in the iid setting, and is critical to ensure robustness when the training data is scarce, skewed, or out-of-domain. the code and pre-trained models are available online 2 . 1 bilingual evaluation understudy with representations from transformers. we refer the intrigued reader to @xcite for a justification of the term understudy. ## fine-tuning bert for quality evaluation given the small amounts of rating data available, it is natural to leverage unsupervised representations for this task. in our model, we use bert (bidirectional encoder representations from transformers) @xcite , which is an unsupervised technique that learns contextualized representations of sequences of text. given x and x, bert is a transformer @xcite that returns a sequence of contextualized vectors: ## pre-training on synthetic data the key aspect of our approach is a pre-training technique that we use to "warm up" bert before fine-tuning on rating data. 3 we generate a large number of of synthetic reference-candidate pairs (z, z), and we train bert on several lexical-and semantic-level supervision signals with a multitask loss. as our experiments will show, bleurt generalizes much better after this phase, especially with incomplete training data. any pre-training approach requires a dataset and a set of pre-training tasks. ideally, the setup should resemble the final nlg evaluation task, i.e., the sentence pairs should be distributed similarly and the pre-training signals should correlate with human ratings. unfortunately, we cannot have access to the nlg models that we will evaluate in the future. therefore, we optimized our scheme for generality, with three requirements. ## experiments in this section, we report our experimental results for two tasks, translation and data-to-text. first, we benchmark bleurt against existing text generation metrics on the last 3 years of the wmt metrics shared task @xcite . we then evaluate its robustness to quality drifts with a series of synthetic datasets based on wmt17. we test bleurt's ability to adapt to different tasks with the webnlg 2017 challenge dataset @xcite . finally, we measure the contribution of each pre-training task with ablation experiments. our models: unless specified otherwise, all bleurt models are trained in three steps: regular bert pre-training @xcite , pre-training on synthetic data (as explained in section 4), and fine-tuning on task-specific ratings (translation and/or data-to-text). we experiment with two versions of bleurt, bleurt and bleurtbase, respectively based on bert-large (24 layers, 1024 hidden units, 16 heads) and bert-base (12 layers, 768 hidden units, 12 heads) @xcite , both uncased. we use batch size 32, learning rate 1e-5, and 800,000 steps for pre-training and 40,000 steps for finetuning. we provide the full detail of our training setup in the appendix. ## conclusion we presented bleurt, a reference-based text generation metric for english. because the metric is trained end-to-end, bleurt can model human assessment with superior accuracy. furthermore, pre-training makes the metrics robust particularly robust to both domain and quality drifts. future research directions include multilingual nlg evaluation, and hybrid methods involving both humans and classifiers.