A Tutorial on Evaluation Metrics used in Natural Language Generation (2021)

Abstract: The advent of Deep Learning and the availability of large scale datasets has accelerated research on Natural Language Generation with a focus on newer tasks and better models. With such rapid progress, it is vital to assess the extent of scientific progress made and identify the areas/components that need improvement. To accomplish this in an automatic and reliable manner, the NLP community has actively pursued the development of automatic evaluation metrics. Especially in the last few years, there has been an increasing focus on evaluation metrics, with several criticisms of existing metrics and proposals for several new metrics. This tutorial presents the evolution of automatic evaluation metrics to their current state along with the emerging trends in this field by specifically addressing the following questions: (i) What makes NLG evaluation challenging? (ii) Why do we need automatic evaluation metrics? (iii) What are the existing automatic evaluation metrics and how can they be organised in a coherent taxonomy? (iv) What are the criticisms and shortcomings of existing metrics? (v) What are the possible future directions of research?

## tutorial content description natural language generation (nlg) encompasses various tasks that require an automatic generation of human-understandable text such as machine translation, abstractive summarization, question answering, data-to-text generation, dialogue systems, etc. each of these tasks has several use-cases with numerous models proposed over the years. the successful application of machine learning and deep learning techniques has transformed the mainstream models for nlg from rule-based systems to data-driven, end-to-end trainable systems. the easier availability of datasets and access to powerful computing resources has led to the wide-spread adoption of these techniques and rapid developments in the field. to track the developments and understand the scientific progress made, these nlg systems need to be evaluated carefully. the ideal way to do so would be to employ expert human evaluators. however, this option would be very time-consuming and expensive, and is thus infeasible. hence the community has settled for automatic evaluation metrics to track scientific progress in this field. automatic evaluation metrics such as bleu @xcite , meteor (banerjee and lavie, 2005), rouge @xcite have been around for several years and are still predominantly used. they have also been readily adopted for newer tasks in nlg such as question generation, image captioning, etc, due to the lack of any other relevant metrics. however, there has been heavy criticism for such an adoption of metrics across tasks, corroborated by their poor correlations with human judgements @xcite @xcite . several new metrics are being proposed to address the shortcomings of the existing ones @xcite . the emerging metrics also explore the idea of using the context provided for the task (such as a document, image, passage, or tabular data, and so on), unlike bleu, meteor, rouge, etc. this has lead to the development of 'context-dependent metrics' alongside the 'context-free metrics'. both the context-free and context-dependent metrics can be categorized based on their underlying technique into trained metrics and untrained (i.e., rule-based/ heuristic-based) metrics. untrained metrics can be further classified depending on whether they are word-based @xcite @xcite @xcite , character-based @xcite , or embedding-based @xcite @xcite @xcite . similarly, trained metrics are sub-categorized depending on whether they need input features (such as precision, recall, number of words in a sentence, etc,) @xcite @xcite or whether they extract the features from the input sentences in an end-toend manner @xcite @xcite @xcite @xcite . in this tutorial, we provide an overview of these different techniques that have been used to formulate automatic evaluation metrics. we also discuss the studies that analyze/inspect these metrics and report their shortcomings. the major criticisms on the metrics include the uninterpretability of the scores @xcite , bias towards specific models @xcite or scores @xcite , and their inability to capture all the nuances in a task @xcite . we conclude by presenting the possible next directions of research in automatic evaluation metrics. ## tutorial structure and schedule outline we plan a 3 hour tutorial based on the following content and associated time estimates.