SESCORE 2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes (2023)

Abstract: Is it possible to train a general metric for evaluating text generation quality without human-annotated ratings? Existing learned metrics either perform unsatisfactory across text generation tasks or require human ratings for training on specific tasks. In this paper, we propose SEScore2, a self-supervised approach for training a model-based metric for text generation evaluation. The key concept is to synthesize realistic model mistakes by perturbing sentences retrieved from a corpus. We evaluate SEScore2 and previous methods on four text generation tasks across three languages. SEScore2 outperforms all prior unsupervised metrics on four text generation evaluation benchmarks, with an average Kendall improvement of 0.158. Surprisingly, SEScore2 even outperforms the supervised BLEURT and COMET on multiple text generation tasks.

## introduction recently, researchers made significant progress in text generation: translation @xcite , structured data-to-text @xcite , dialogue generation @xcite , and summarization @xcite . automatic metrics are essential for the development of text generation models as they replace expensive human labor and are able to evaluate the generation performance @xcite , as well as guide the generation process @xcite . how can we efficiently and effectively train a metric for general text generation tasks? depending on the inputs, we can categorize evaluation metrics into source-based, hybrid-based, and reference-based metrics. source-based metrics estimate text quality through the source and are useful when reference is noisy or unavailable @xcite , but they may produce sub-optimal results and explore spurious correlations @xcite . referencebased metrics, when paired with high-quality references, can reflect text generation quality, regardless of source modalities (e.g audio and triples). hybrid metric comet @xcite uses both source and reference. in this work, we aim to construct a reference-based metric, as it is invariant to the source modality, making it suitable for use across various tasks. although learned metrics have been shown to be more effective than rule-based metrics (e.g bleu @xcite ), they still have limitations in terms of evaluation capability and applicability to specific tasks. supervised metrics such as bleurt @xcite and comet @xcite are superior in evaluation, but restricted to tasks with human ratings of generated text. unsupervised metrics, such as bertscore @xcite and bartscore @xcite , do not require human ratings for training, but their correlation with human judgment on specific tasks is still inferior compared to the best supervised metrics @xcite . our goal of this paper is to devise a referencebased automatic evaluation metric that 1) can be learned without a human quality score, 2) align well with human judgments, and 3) can be generalized to a wide variety of domains and nlg tasks. to achieve this, we propose a self-supervised training method using text with synthetic mistakes. our main intuition is that these synthetic mistakes should contain errors at different severity levels and appear realistic ("realistic mistakes" are defined as natural and model output-like mistakes). we are inspired by the human evaluation protocol mqm @xcite which assesses translation quality by identifying errors with two levels of severity. to make the synthetic mistakes realistic, we mine surface differences among groups of similar sentences and use these different phrases to perturb the original text and construct mistakes (see figure 2 ). unlike previous methods that utilize generative models to synthesize errors @xcite , our approach employs retrieved similar sentences, making it more general. to encompass text diversity, anchor texts are sampled from large-scale parallel corpora. additionally, a novel pretraining signal is proposed to train sescore2, which aims to resemble the way humans grade model outputs by estimating the severity levels of each mistake @xcite . to date 2 , we support six languages: english, german, chinese, japanese, russian, spanish. the primary advantage of the sescore2 is its ease of extension to numerous other languages while providing reliable severity estimation. our contributions to this paper are as follows: ## related work human evaluation metrics such as crowd-worker evaluation using direct assessment (da) are widely used in wmt shared task competition @xcite @xcite . @xcite ; @xcite find that crowdworkers fail to discriminate human and machine outputs. @xcite improves human ratings by using multidimensional quality metrics (mqm) framework @xcite with language experts. each annotated error can be categorized into multiple types and is associated with different severity levels, such as major and minor. automatic evaluation metrics such as rule-based metrics (e.g. n-gram matching bleu @xcite , chrf @xcite ) and distancebased (e.g. ter @xcite ) have been commonly used in text generation evaluations because they are fast and domain invariant. however, they have limitations in capturing semantics and long-distance dependencies @xcite . the supervised learned metrics @xcite are directly optimized from human ratings. however, they may have poor generalization to unseen domains and tasks @xcite . unsupervised learned metrics attempt to obtain training objectives other than human ratings @xcite @xcite . however, as pointed out by @xcite , they are limited on the error types that they can evaluate (e.g accuracy) and can not go beyond (e.g fluency or style). some recent studies attempt to mitigate this issue by generating synthetic data via paraphrasing and perturbations @xcite @xcite . to further derive the fine-grained pretraining signals, sescore @xcite leverages language models to generate multiple error types in one segment and estimate each error's severity level. however, model-dependent data synthesis can intrinsically introduce model bias and limit the diversity of data samples. sescore2 develops a novel retrieval augmented synthesis technique, which is task-agnostic and can simulate diverse and realistic model mistakes using the parallel corpora. inspired by @xcite , we obtain our pretraining signals aligning with human grading process. ## problem definition the task is to estimate the quality score between a reference and a model-generated hypothesis. follow @xcite , if the model error semantically alters the meaning of the sentence, we label it as major, otherwise as minor. see the example in figure 1 , where a 4-point star represents the reference, and circles and triangles represent the sentences with minor and major mistakes, respectively. a minor mistake contributes a -1 score, and a major mistake contributes a -5 score. the severity score of a hypothesis is the sum of all minor and major contributions and is no less than -25. given a set of reference, hypothesis, humanannotated severity score triples (x, y, s), our goal is to train a learned metric, m (x, y) → s. due to the scarcity of human ratings, such triples are unavailable for most tasks. thus, we propose an automatic way to generate synthetic training samples. ## the sescore2 approach sescore2 is a ssl technique, initialized with pretrained embeddings, like bert, then trained with task-agnostic nlg evaluation objective on large-scale synthetic data. no specific fine-tuning is required at inference time evaluation. there are three steps: 1) data construction that samples source-target pairs (t, x) from machine translation (mt) corpora and creates synthetic text y from x using retrieval augmented synthesis; 2) severity score labeling that automatically generates label s using (t, y); 3) model training which pretrains a regression model m using triples (x, y, s). during inference, sescore2 only takes reference and model output as input and estimates the quality score, which can be applied to different text generation tasks. detailed implementations of the model can be found in appendix 5. ## experiments to verify the generalization ability of the sescore2, we investigate the following questions: corresponding to the aforementioned evaluation aspects: ## conclusion we propose a novel retrieval augmented synthesis method for generating diverse and realistic errors on a large scale, with varying severity levels. our experiment demonstrates, sescore2, with its human-aligned self-supervised objective, can outperform prior metrics or match supervised metrics in four text generation tasks across three languages. lastly, we demonstrate sescore2 can correlate well with multiple evaluation aspects, such as fluency and quality.