Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021)

Abstract: Natural language generation (NLG) spans a broad range of tasks, each of which serves for specific objectives and desires different properties of generated text. The complexity makes automatic evaluation of NLG particularly challenging. Previous work has typically focused on a single task and developed individual evaluation metrics based on specific intuitions. In this paper, we propose a unifying perspective based on the nature of information change in NLG tasks, including compression (e.g., summarization), transduction (e.g., text rewriting), and creation (e.g., dialog). _Information alignment_ between input, context, and output text plays a common central role in characterizing the generation. With automatic alignment prediction models, we develop a family of interpretable metrics that are suitable for evaluating key aspects of different NLG tasks, often without need of gold reference data. Experiments show the uniformly designed metrics achieve stronger or comparable correlations with human judgement compared to state-of-the-art metrics in each of diverse tasks, including text summarization, style transfer, and knowledge-grounded dialog.

## introduction natural language generation (nlg) refers to the broad set of tasks that produce fluent text from input data and other contextual information. the diverse tasks serve for vastly different uses in practice. for example, summarization compresses a source article into a short paragraph containing the most important information; translation transduces content expressed in one language into another; and a chatbot creates novel responses to drive the conversation. recent years have seen remarkably fast progress in improving and making new models 1 code available at https://github.com/ tanyuqian/ctc-gen-eval for nlg tasks. however, evaluation of nlg has long been considered difficult @xcite : human evaluation is often prohibitively expensive and slow, while accurate automatic evaluation is challenging given the complexity of text modeling and the diverse aspects to be measured for different nlg tasks. previous work has developed a large variety of automatic metrics. a popular general strategy is to measure the similarity of generated text against human-written references, such as the classical bleu @xcite , rouge @xcite , and more recent variants based on neural models (e.g., @xcite . however, an nlg task typically involves multiple desirable properties (e.g., consistency, conciseness, richness) that may have different priorities and need trade-off depending on the application scenarios @xcite @xcite . thus a single score without multi-aspect interpretability is often inadequate to characterize generation quality. a growing number of recent works have proposed aspect-based metrics for popular tasks such as summarization @xcite and dialog @xcite . those metrics are typically each designed for individual tasks and aspects, based on specific intuitions. the lack of a common theoretical ground makes it difficult to share the evaluation strengths across the diverse nlg problems, and fails to offer guidance to metric design for emerging tasks and aspects. in this paper, we propose a more unifying perspective of nlg evaluation through the lens of information change, which offers a general framework to measure many key aspects of nlg tasks. in particular, based on the practical use of nlg, each task can be seen as one of ( 1 ) compression to express salient information in concise text, such as summarization and image captioning; (2) transduction to transform text while preserving content precisely, such as translation and style transfer; and inspired by recent work on model-based evaluation, we adopt contextualized language models to measure information alignment. we then demonstrate the framework by devising a family of highly intuitive metrics for three representative tasks (aspects) in each category, respectively, including summarization (relevance and consistency), style transfer (content preservation) and knowledge-based dialog (engagingness and groundedness). experiments show that the uniformly designed metrics robustly outperform or compete with state-of-theart metrics specifically designed for each task, in terms of correlations with human judgement. we also study different implementations of the central information alignment estimation model, showing that improved alignment measure leads to better evaluation quality across all the tasks/aspects. several emerging nlg benchmarks @xcite collected existing metrics for various tasks, whereas we aim at developing new unified metrics with stronger performance. @xcite proposed a categorization for different nlg quality aspects. our general framework covers all the described types of quality. ## a unified evaluation framework we present the new framework that offers a common foundation for characterizing diverse nlg tasks and leads to a set of interpretable metrics for evaluating their key aspects. as discussed in §1, nlg tasks can be categorized as performing compression, transduction, or creation based on changes in conveyed information from input to output. for a compression task (e.g., summarization), the goal is to concisely describe the most important information in the input (e.g., a document). that is, the output should only contain content from the input, namely "consistency" @xcite @xcite , and the included content must be salient, namely "relevance" @xcite . intuitively, with an "information alignment" measure that assesses how the information in a generated output overlaps with that in the input (and in references that offer clues for salience), we can readily evaluate the two key aspects. the same intuition applies to transduction tasks (e.g., style transfer), where the output must preserve the input content precisely. the evaluation of "preservation" @xcite thus also boils down to measuring the information alignment between input and output. a creation task (e.g., dialog) generates output that adds on top of input (e.g., dialog history) new information (e.g., from external knowledge). information alignment between the output, input, and external sources is thus essential for evaluating how well the created content engages with the context @xcite and how meaningful the content is by grounding to the external sources @xcite . from the above perspective, information alignment arises as a common central component that connects evaluations across the tasks. a single accurate alignment prediction model would enable us to reliably evaluate many relevant aspects in various applications. next, we first present our definition of information alignment ( §3.1); then describe the details of how the aspect metrics for compression, trans-duction, and creation are built on the alignment ( §3.2-3.4); we finally discuss different effective implementations of the underlying alignment estimation model based on neural networks ( §3.5). ## experiments we evaluate the proposed metrics on commonly used human annotation datasets for summarization ( §4.1), style transfer ( §4.2) and dialog ( §4.3), and study the effect of information alignment accuracy on the performance of metrics ( §4.4). ## conclusions we have proposed a general evaluation framework for nlg tasks categorized as compression, transduction, and creation. based on the concept of information alignment between input, context, and output, we devised a family of interpretable metrics for the key aspects of diverse tasks (summarization, style transfer, and dialog). the uniformly designed metrics achieve superior or comparable human correlations compared to existing metrics. the unified framework offers a structured guidance for the metric design of new aspects/tasks, which we are excited to explore more in the future.