MISMATCH : Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types (2023)

Abstract: With the growing interest in large language models, the need for evaluating the quality of machine text compared to reference (typically human-generated) text has become focal attention. Most recent works focus either on task-specific evaluation metrics or study the properties of machine-generated text captured by the existing metrics. In this work, we propose a new evaluation scheme to model human judgments in 7 NLP tasks, based on the fine-grained mismatches between a pair of texts. Inspired by the recent efforts in several NLP tasks for fine-grained evaluation, we introduce a set of 13 mismatch error types such as spatial/geographic errors, entity errors, etc, to guide the model for better prediction of human judgments. We propose a neural framework for evaluating machine texts that uses these mismatch error types as auxiliary tasks and re-purposes the existing single-number evaluation metrics as additional scalar features, in addition to textual features extracted from the machine and reference texts. Our experiments reveal key insights about the existing metrics via the mismatch errors. We show that the mismatch errors between the sentence pairs on the held-out datasets from 7 NLP tasks align well with the human evaluation.

## introduction large language models have pushed the boundaries for natural language generation (nlg). more and more, the generated machine texts look human-like. the need for evaluation metrics has never been so critical in the recent decade. typically, there are two ways to evaluate the quality of machinegenerated text: automatic evaluation and human evaluation. in automatic evaluation, the quality of the machine-generated text is captured using a single number from a range of values indicating how good the generated text is by a (hand-coded rule-based or neural-based) model. several nlp tasks still use the metrics from 2 decades ago, for instance, @xcite and meteor @xcite for abstractive summarization, bleu @xcite for machine translation, etc. it has been noted in several works that automatic evaluation metrics are incapable of capturing the different criteria in measuring the quality of the text and often have a poor correlation with human judgments @xcite . the current automatic evaluation metrics lack the ability to measure the quality of a modern machine-generated text. in human evaluation, we evaluate the machine text based on human ratings, where we ask human annotators to judge a given pair of texts. the quality of the machine text is measured using different task-specific human evaluation criteria such as fluency, coherence, correctness, consistency, relevance, adequacy, etc. human evaluations are often expensive, time-consuming, and subjective (low inter-annotator agreement), especially when broad criteria such as the fluency of the generated text and the interestingness of the model-generated text are used for human judgment. to address these challenges in automatic and human evaluations, there have been recent efforts in the fine-grained evaluation of generated text in several nlp domains @xcite @xcite . in this paper, we are interested in utilizing fine-grained evaluation categories to guide the prediction of human judgments. towards this goal, we introduce a task-agnostic list of 13 mismatch error types, such as grammatical errors, spatial/temporal errors, etc, that unifies several related task-specific efforts @xcite @xcite . these mismatch error types are comprehensive, interpretable, and useful for predicting human evaluation criteria. for example, an occurrence of grammatical error in a machine-generated text can impact its fluency rating. figure 1 gives the overview of the proposed mismatch error types for fine-grained evaluation. we propose a neural framework for evaluation that uses these mismatch error types as auxiliary tasks to model the human judgment and repurposes automated evaluation metrics as additional scalar features, concatenated to textual features extracted from the machine and reference texts via pre-trained lm text embeddings @xcite . we show that pre-training our proposed model using synthetic data for the mismatch prediction task, and fine-tuning using real data for human evaluation criteria, for different nlp tasks, achieves state-of-the-art performance on the main downstream task of predicting human evaluation metrics. we provide several ablation studies showing the importance of each component of our architecture, and the correlations between the mismatch error types and the automatic and human evaluation metrics. we also show how our architecture is useful in predicting novel evaluation criteria, such as factuality in abstractive summarization. ## nlg evaluation given a pair of texts: a reference text and a machine-generated one, we are interested in evaluating the quality of the generated text using the reference text. we measure the quality of the generated text by estimating how a human will judge this text based on different evaluation criteria. such evaluation is common in many ml/nlp tasks, e.g., machine translation, summarization, image captioning, etc. unlike in other automatic evaluation metrics, we consider fine-grained evaluation cues from 13 mismatch error types, inspired by several related task-specific efforts @xcite @xcite to guide the main task of predicting the human judgments. we propose a neural framework for evaluating machinegenerated texts that use these mismatch error type predictions as auxiliary tasks, and automated evaluation metrics as additional scalar features, along with the pair of pre-trained lm text embeddings extracted from reference and generated texts. in this section, we discuss the role of mismatch error types as a good proxy for human judgments (section 2.1) and the model architecture for the proposed approach (section 2.2). ## experimental results in this section, we show experimental results validating the proposed model for predicting human judgments. ## related work automatic evaluation metrics such as rouge @xcite , bleu @xcite , me-teor @xcite , have been proposed for different tasks as a substitute for human annotations. in nlp, text generation tasks have extensively used these metrics to measure the quality of the machine-generated text. evaluation metrics are either task-specific (anli @xcite , summac @xcite , cider @xcite , supert @xcite ) or task-agnostic (bertscore @xcite , bleurt @xcite ), and are based on either human handcrafted logic (rouge, bleu, meteor) or neural framework (bertscore, bleurt). for human annotation, several dimensions such as coherence, consistency, and fluency are considered to measure the quality of the generated text, yet most evaluation metrics compute a single score to summarize the evaluation. further, these single-scored metrics often do not correlate well with the human ratings. to address this problem, several attempts have been proposed to combine multiple evaluation metrics. rose @xcite uses a linear combination of rouge variations (rouge_1, rouge_2, rouge_l, rouge_lsum) for the machine translation task and the combined score is better than the individual rouge scores in the evaluation. s 3 @xcite uses the combination of the rouge scores and jenson-shannon divergence to predict the human rating. neural-based approaches such as bleurt and comet directly train on the overall human rating for the sample texts. even though neural-based evaluation metrics seem promising, they often require tens of thousands of training samples to mimic human rating, struggle with new domains/tasks and unseen samples, and still output a single score. understanding evaluation metrics: recently, there has been growing interest in understanding what these evaluation metrics measure in terms of fine-grained evaluation criteria. it is done by studying different error categories (mismatches) in the machine-generated texts but claim none of the existing evaluation metrics can predict all of the mismatches, without providing any solution. perturbation checklist @xcite uses template-based perturbation on multiple tasks based on the human evaluation criteria to study mismatches. frank @xcite studies different evaluation metrics on factuality in abstractive summarization using error types. scarecrow @xcite explores errors in prompt-based text generation by large language models. breakingnli @xcite evaluates different metrics on synthetic data created from the external knowledge graph wordnet @xcite . @xcite studies different types of factuality and hallucinations in the generated text by large language models. in this work, we unify these task-specific research directions. we propose several evaluation models that combine the best of handcrafted logic on robust evaluation, a neural framework for text representations, and mismatch error types types to measure the quality of the generated text based on the human evaluation criteria. ## conclusion in this paper, we proposed a neural framework for evaluating the quality of the machine-generated text w.r.t the reference text. to achieve this, we defined a set of mismatch error types to approximate the human ratings over a set of evaluation criteria. we showed that in addition to the bert-based text representation, feature-invariant representations learned from the automatic evaluation metrics improve the prediction of both the mismatch types as well as human ratings with pre-training on only a limited amount of synthetic examples with mismatch error types. we further showed that mismatches between pairs of texts provide an interpretable way to explain human judgments, through a series of ablation studies and correlation analyses. our proposed mismatch error types is a crucial bridge between automatic evaluation metrics and human evaluation criteria, leading to more interpretable predictions for nlp models.