Toward Human-Like Evaluation for Natural Language Generation with Error Analysis (2023)

Abstract: The pretrained language model (PLM) based metrics have been successfully used in evaluating language generation tasks. Recent studies of the human evaluation community show that considering both major errors (e.g. mistranslated tokens) and minor errors (e.g. imperfections in fluency) can produce high-quality judgments. This inspires us to approach the final goal of the automatic metrics (human-like evaluations) by fine-grained error analysis. In this paper, we argue that the ability to estimate sentence confidence is the tip of the iceberg for PLM-based metrics. And it can be used to refine the generated sentence toward higher confidence and more reference-grounded, where the costs of refining and approaching reference are used to determine the major and minor errors, respectively. To this end, we take BARTScore as the testbed and present an innovative solution to marry the unexploited sentence refining capacity of BARTScore and human-like error analysis, where the final score consists of both the evaluations of major and minor errors. Experiments show that our solution consistently and significantly improves BARTScore, and outperforms top-scoring metrics in 19/25 test settings. Analyses demonstrate our method robustly and efficiently approaches human-like evaluations, enjoying better interpretability. Our code and scripts will be publicly released in https://github.com/Coldmist-Lu/ErrorAnalysis_NLGEvaluation .

## introduction leveraging the power of large pre-trained language models (plms) has been proven effective in evaluating natural language generation (nlg) tasks @xcite . metrics like bertscore @xcite and mover- recent studies of the human evaluation community show that the quality of human judgments can be improved through fine-grained error analysis, incorporated in an error-based framework multidimensional quality metric (mqm) @xcite . mqm requires evaluators to identify errors and categorize them into different levels according to their severity. for instance, mistranslations @xcite and hallucinations @xcite are mostly considered as major errors, and imperfections in fluency @xcite are often marked as minor errors. different weights are then assigned to major/ minor errors, resulting in high-quality human evaluation scoring. analogous to major/ minor errors in mqm, we take the first step to consider incorporating the evaluation of explicit/ implicit errors into plmsbased metrics. specifically, we use bartscore, a state-of-the-art metric for nlg by @xcite as the test bed, and propose a metric called bartscore++. we present an overview of our proposed method in figure 1 . in particular, given the hypothesis and reference, we propose an error analysis framework to obtain a refined sentence (see example in we experiment on machine translation (mt), text summarization (sum), and data-to-text (d2t), and show that bartscore++ consistently and significantly improves the performance of vanilla bartscore, and surpasses existing top-scoring metrics in 19 out of 25 test settings, even exceeding human performance on summarization dataset rank19. we give further analyses to confirm that the consistent improvements come from the humanlike (specifically, mqm-like) error judgment. our main contributions are as follows: besides taking bartscore as the testbed to verify the effectiveness of our proposed error-analysis evaluation strategy, we also show the universality in the recently advanced language model chatgpt foot_0 by designing an error-analysis-based prompt @xcite . we anticipate that our strategy will 2 preliminaries problem formulation the goal of nlg evaluation is to acquire a score measuring the quality of generated text y given a reference signal r. unless otherwise stated, r represents the sentence properly created by human experts to assist in evaluation, and @xmath0. . . y n ), called hypothesis in this paper, refers to the generated text to be evaluated foot_1 . bartscore bartscore is a sota metric proposed by @xcite for universal nlg evaluation. the idea of bartscore is to utilize the generation probabilities of a large pre-trained model bart @xcite to measure the quality of sentences. it autoregressively computes the log probabilities of each token in the hypothesis, and then averages them as the overall score. this evaluation process can be formally written as: based on this formulation, bartscore creates specific variants for different evaluation scenarios. we summarize their usage in appendix a. for simplification, we use the notation of barts(y, r) when vanilla bartscore is further applied. mqm mqm is an error-based human evaluation framework, which is commonly agreed to be more reliable than traditional human evaluation techniques @xcite . in mqm framework, each evaluator is asked to identify all errors in a sentence and categorize them into major and minor levels indicating their severities. sentences will be marked an non-translation error if they are not possible to reliably identify errors. major/ minor errors are then assigned with different weights, and the final mqm score is computed through the weighted sum of errors @xcite . inspired by the mechanism of mqm, we take a step toward human-like evaluation by incorporating error analysis into bartscore. ## methodology to better understand how bartscore++ works, we show a running example of our method in figure 1 . ## analysis to better understand the mechanism by which bartscore++ achieves promising results, we take a closer look and answer four questions: q1: how reliable is our bartscore++ when evaluating top-performing systems? q3: how does error analysis make bartscore++ more human-like? q4: does error analysis framework introduce significant latency? for mt evaluation in this section, we use mqm, an error-based evaluation framework annotated by human experts @xcite . for a fair comparison, the error weight ratio λ for wmt20 zh-en test set is fixed to 1.7. bartscore++ is reliable when evaluating top-k systems previous studies have shown that most metrics are unreliable for evaluating bestperforming systems, showing a sharp degradation of correlation with human evaluation @xcite ). to answer q1, we assess our method shown in figure 3 with several baseline metrics on top-k mt systems by computing kendall's τ respectively. as seen, bartscore++ can further improve bartscore's performance, especially when evaluating top-performing systems (@xmath1. this verifies the reliability of our purposed method. 2 4 6 8 10 no. of k in top-k sampling 0.0 0.5 1.0 1.5 2.0 2.5 latency (s/sent) realsumm bartscore++ w/o prompt bartscore++ w/ prompt bartscore w/o prompt bartscore w/ prompt iteration 1 2 3 4 5 6 iteration figure 5: time efficiency analysis on summarization task (realsumm). we report the average latency for evaluation and the average number of iterations w.r.t. top-k sampling in correct stage in §3.2. ## conclusion we present an automatic metric bartscore++ for nlg evaluation. inspired by the advanced human evaluation mqm, bartscore++ incorporates error analysis strategies to give a comprehensive score considering explicit and implicit errors. experimental results show our approach achieves competitive results on a broad range of tasks. our work is an early step toward humanlike evaluation for automatic metrics, and we hope our bartscore++ can motivate researchers working on nlg evaluation to focus more on human evaluation procedures such as error analysis.