Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022)

Abstract: Is it possible to build a general and automatic natural language generation (NLG) evaluation metric? Existing learned metrics either perform unsatisfactorily or are restricted to tasks where large human rating data is already available. We introduce SESCORE, a model-based metric that is highly correlated with human judgements without requiring human annotation, by utilizing a novel, iterative error synthesis and severity scoring pipeline. This pipeline applies a series of plausible errors to raw text and assigns severity labels by simulating human judgements with entailment. We evaluate SESCORE against existing metrics by comparing how their scores correlate with human ratings. SESCORE outperforms all prior unsupervised metrics on multiple diverse NLG tasks including machine translation, image captioning, and WebNLG text generation. For WMT 20/21En-De and Zh-En, SESCORE improve the average Kendall correlation with human judgement from 0.154 to 0.195. SESCORE even achieves comparable performance to the best supervised metric COMET, despite receiving no human annotated training data.

## introduction text generation tasks such as translation and image captioning have seen considerable progress in the past few years @xcite . however, precisely and automatically evaluating generated text quality remains a challenge. long-dominant n-gram-based evaluation techniques, such as bleu @xcite and rouge @xcite , are sensitive to surfacelevel lexical and syntactic variations, and have been repeatedly reported to not well correlate to human judgements @xcite . multiple learned metrics have been proposed to better approximate human judgements. these met-rics can be categorized into unsupervised and supervised methods based on whether human ratings are used. the former includes prism @xcite , bertscore @xcite , bartscore @xcite , etc. the latter includes bleurt @xcite , comet @xcite etc. unsupervised learned metrics are particularly useful as task-specific human annotations of generated text can be expensive or impractical to gather at scale. while these metrics are applicable to a variety of nlg tasks @xcite , they tend to target a narrow set of aspects such as semantic coverage or faithfulness, and have limited applicability to other aspects, such as fluency and style, that matter to humans @xcite . while supervised metrics can address different attributes by modeling the conditional distribution of real human opinions, training data for quality assessment is often taskand domain-specific with limited generalizability. we introduce sescore, a general technique to produce nuanced reference-based metrics for automatic text generation evaluation without using human-annotated reference-candidate text pairs. our method is motivated by the observation that a diverse set of distinct error types can co-occur in candidate texts, and that human evaluators do not view all errors as equally problematic @xcite . to this end, we develop a stratified error synthesis procedure to construct (reference, candidate, score) triples from raw text. the candidates contain non-overlapping, plausible simulations of nlg model errors, iteratively applied to the input text. at each iteration, a severity scoring module isolates individual simulated errors, and assesses the human-perceived degradation in quality incurred. our contributions are as follows: ## related work traditional n-gram matching based @xcite and edit distance based approaches @xcite have proven to be limited in recognizing semantic similarity beyond the lexical level. learned metrics @xcite @xcite have been proposed to align better with human judgements. we categorize these metrics as either unsupervised or supervised with respect to learning from human-annotated scores. unsupervised metrics attempt to extract features from large pretrained models. embeddingbased metrics (e.g. bertscore @xcite and moverscore @xcite ) create soft-alignments between reference and hypothesis in the embedding space. however, they are refined in the semantic coverage. text generationbased metrics @xcite , use conditional probability of the generated sentence to evaluate faithfulness of the candidates. however, @xcite points out text generation evaluation can produce errors beyond semantic coverage or faithfulness (e.g. style and fluency errors), which results poor correlations to the human evaluations. supervised metrics attempt to learn through limited human-labelled severity annotations. @xcite trained comet on a small set of domainspecific human ratings; this model has limited extensibility to teh general domain. bleurt @xcite first pretrains on millions of synthetic data and then uses wmt testing data in fine-tuning the model. unlike our fine-grained stratified error synthesis, the labels on the synthetic data are derived from prior metrics or other tasks, limiting the quality and precision of pretraining process. ## the sescore approach given a reference text x x x and a candidate y y y, a metric is expected to output a score s. training such a metric model requires triples of reference-candidatescore's. however, there are no large-scale human transformer layers pooling layer sentence embedding features feedforward nn sescore(x,y') raw text (x) synthesized text (y') synthetic quality score (s') m s e lo s s reference (x) candidate (y) pre-training stage inference stage transformer layers pooling layer sentence embedding features feedforward nn sescore(x,y') annotated triple data available in many tasks. we consider a general setup where large raw text corpus is available. sescore is trained from a pretrained language model (e.g. bert) on synthetic triples generated from raw text. it synthesizes candidate sentences y y y ′ to mimic plausible errors by transforming raw input sentences x x x multiple times. at each step, it inserts, deletes, or substitutes a random span of text. these errors are non-overlapping. it assesses the severity of the errors introduced in the transformation. this allows us to pretrain quality prediction models on corpora containing only raw text samples {x}, enabling the use of learned quality prediction models in any text generation domain. the process of generating y y y ′ from x x x, stratified error synthesis, is so called for its incremental and multi-category nature; a stochastic perturbation function g es which randomly samples from a set of potential errors is recursively applied on x x x (eq. ( 1 )) m times to produce a sequence of perturbed sentences z z @xmath0.2). the stratum sentence sequence z is then used to in the subsequent severity scoring step which uses a pairwise severity scoring function s es on consecutive pairs and cumulatively yield training labels s ′ = m @xmath1, z i ) ( § 3.3). a concrete example is illustrated in fig. 2 . finally, we train sescore's quality prediction model, f θ ( fig. 1 ) using synthetic {⟨x x x, y y y ′ , s ′ ⟩} triples ( § 3.4). ## experiments we conduct experiments on three tasks: machine translation, data-to-text and image captioning, to verify the utility and generalizability of sescore. specifically, we compare sescore on wmt 2020 and 2021 test sets in english-to-german (en-de) and chinese-to-english (zh-en) with mqm labels @xcite , which consists of expert-labeled scores. for data-to-text, we test sescore on the webnlg 2017 challenge @xcite . for image captioning, we test sescore on the coco image captioning challenge 2015 @xcite . we use @xcite annotated ted dataset as our development set to select the hyper-parameters in error synthesis models and sescore metric model. we comprehensively analyze each component of our pipeline and their contributions to the final results. ## quantitative analysis to validate the proposed sescore training technique, we analyze the effects of data quantity, the stratified components, and synthetic error types. we include the cross-domain evaluation in the appendix d.1. we include a detailed qualitative analysis of sescore regarding to its robustness and limitations in appendix e. ## conclusion to conclude, we introduced sescore, a referencebased metric for text generation evaluations. without human labels, sescore can outperform all unsupervised evaluation metrics and achieve com-petitive performance to the sota supervised approaches. we demonstrate that our stratified error synthesis approach makes model aware of individual errors with different severity levels, achieving high correlation to the human judgements.