NLG -Metricverse: An End-to-End Library for Evaluating Natural Language Generation (2022)

Abstract: Driven by deep learning breakthroughs, natural language generation (NLG) models have been at the center of steady progress in the last few years, with a ubiquitous task influence. However, since our ability to generate human-indistinguishable artificial text lags behind our capacity to assess it, it is paramount to develop and apply even better automatic evaluation metrics. To facilitate researchers to judge the effectiveness of their models broadly, we introduce NLG-Metricverse—an end-to-end open-source library for NLG evaluation based on Python. Our framework provides a living collection of NLG metrics in a unified and easy-to-use environment, supplying tools to efficiently apply, analyze, compare, and visualize them. This includes (i) the extensive support to heterogeneous automatic metrics with n-arity management, (ii) the meta-evaluation upon individual performance, metric-metric and metric-human correlations, (iii) graphical interpretations for helping humans better gain score intuitions, (iv) formal categorization and convenient documentation to accelerate metrics understanding. NLG-Metricverse aims to increase the comparability and replicability of NLG research, hopefully stimulating new contributions in the area.

## introduction natural language generation (nlg) is a sub-field of natural language processing (nlp) concerned with automatically generating human-understandable text from input data, like prompts, tables, graphs, and images. remarkably, the ability of a machine to produce text indistinguishable from that written by humans is a pre-requisite for artificial general intelligence (agi)-the holy grail of ai. recent advancements in deep learning have yielded tremendous improvements in the nlp sector, making a set of automatic metrics is selected to build a scorer object, concurrently appliable to contexts, predictions, and references with arbitrary n-arity. a meta-evaluation module allows to inspect metrics' performance on the input data or standard benchmarks. finally, a visualization module can be applied to overcome opacity and understand metric-specific scoring processes. nlg the object of fast-growing interest from the research community, as aptly demonstrated by gpt-3 @xcite . pre-trained language models with transformer-based architectures @xcite continue to push the envelope with unprecedented performance and encourage more and more applications. indeed, today nlg includes a wide variety of tasks, such as machine translation, single/multi-document summarization, data-to-text, text-to-text, dialogue generation, free-form question answering, and image/video captioning @xcite . as nlg models get better over time, accurately evaluating them is becoming an increasingly pressing priority for tracking progress in the area and convincingly recognizing state-of-the-art systems. however, the assessment of nlg model output is notoriously a challenging problem @xcite . it involves the consideration of multiple intrinsic quality dimensions (e.g., informativeness, fluency, coherence, adequacy) and open-ended scenarios, where different plausible or equal-meaning responses may exist for the same user input. human evaluation is typically regarded as the gold standard. nevertheless, designing crowdsourcing experiments accompanied by elaborated guidelines is an expensive and high-latency process, which does not easily fit in a daily model development pipeline with the need for automatic benchmarking and tuning at scale. furthermore, as nlg models improve, evaluators are asked to read longer passages of text conditioned on large amounts of context. in these cases, errors are often content-based (e.g., factual inaccuracies or context inconsistencies) rather than fluency-based, making superficial reads and non-expert annotators insufficient @xcite . given these issues, nlg researchers have settled for automatic evaluation metrics computing a holistic or dimension-specific score, an acceptable proxy for effectiveness and efficiency. unfortunately, despite the rapid surge of machinegenerated language, evaluation metrics have fallen behind, leaning on the conservative use of surfacelevel lexical similarities, which fail to cope with diversity and capture the text's underlying meaning. to overcome this severe bottleneck, the community has witnessed-in a relatively short time-a prolific, variegated, and original research production. new nlg metrics are constantly being proposed in top conferences, exhibiting one or more of the following characteristics: (i) use of contextualized word embeddings @xcite , (ii) pre-training on massive unlabeled corpora @xcite , (iii) fine-tuning on data annotated with human judgments @xcite , (iv) management of task-specific nuances @xcite . per contra, nlg metrics today are often designed and implemented from scratch with distinct environments, assumptions, properties, settings, benchmarks, and features. such heterogeneity and disgregation make them difficult to compare or move to slightly different contexts. concretely, the absence of a collective and continuously updated repository-well-documented and covering the entire nlg evaluation pipeline-discourages the use of modern solutions and slows down their understanding and practical application. such barrier is highlighted also by the latest surveys @xcite . in the quest to fill this gap, we present nlg-metricverse foot_1 , an open-source (mit licensed) end-to-end library for nlg evaluation, devised to provide a shared and collaborative codebase for fast application, analysis, comparison, visualization, and prototyping of automatic metrics. the rest of the paper is organized as follows. first, we enumerate the design principles at the basis of nlg-metricverse ( §3), clarify the context, and summarize prior work related to this project ( §2). then, we describe the overarching nlg evaluation framework that constitutes the conceptual foundation for our contributions ( §4). next, we examine the main modules of the library: metrics, meta-evaluation, and visualization ( §5). lastly, we close the discussion and point out possible extensions ( §7). ## background and related work early lexical nlg metrics, such as the bleu @xcite and rouge @xcite , still appear to dominate the landscape, waiting for feasible, robust, and widely-adopted alternatives. despite the high number of criticisms and studies proving their poor correlation with human judgment @xcite , the popularity of firstgeneration metrics has not declined but expanded with the emergence of deep neural networks and new tasks. simplicity, consistency, unsupervision, lightweight, and fast computation are the central basis of this success. however, it has become increasingly clear that such adoption is often not prudent. metrics measuring surface-level overlap are unsuitable for advanced evaluation, especially for modern text generation systems trained on mammoth data and with impressive paraphrasing capabilities @xcite )-where ideal metrics should be sensitive to the underlying semantics. as a remedy, nlg researchers have started injecting learned/learnable components into their metrics, moving from a discrete space of word tokens to a continuous highdimensional space of word vectors, thereby capturing distributional semantics. over the years, many strong nlg evaluation metrics have been proposed, particularly transformer-based, like bleurt (sel-lam et al., 2020), bertscore @xcite , and bartscore @xcite . the trend towards the definition of model-based metrics and the resolution of task-specific needs have created a fertile ground for research. according to @xcite , from 2002 (when bleu was proposed) to 2014 (when deep learning became prevalent), there were only about 10 automatic nlg evaluation metrics in use; since 2015, at least 36 new metrics have appeared. on the other side, metrics are often scattered online, non-maintained, undocumented, implemented in various languages, inconsistent with the paper results. this not only hampers reproducibility but also inhibits scalability, as each research paper ends up creating its own implementation almost from scratch. some libraries have already tried to make an integrated environment. to our best knowledge, nlgeval @xcite , hugginface datasets @xcite , evalaute foot_3 , torch-metrics (detlefsen et al., 2022), and jury (cavusoglu et al., 2022) are the only resources available. however, none of them possess all the properties listed below: (i) large number of heterogeneous nlg metrics, (ii) concurrent computation of more metrics at once, (iii) support for multiple references and/or predictions, (iv) meta-evaluation, and (v) visualization. ## design principles nlg-metricverse has been designed with five main principles in mind, which, we argue, can help researchers and practitioners in a number of ways. comprehensiveness given the impressive pace at which the field is growing, comprehensiveness is imperative, with the ultimate goal of providing a unique, smooth, and up-to-date access point to all the most relevant nlg evaluation metrics disseminated in different streams of literature. we also comprise organization and consistency across the library, with a coherent interaction between modules and sub-modules. this principle revolves around consolidating an all-in-one communitydriven library, integrating ready-to-use n-gram-and embedding-based metrics-supervised and unsupervised, trained and untrained, reference-and statistics-based, task-specific and general-purpose, sentence-and document-level. from this synergy, we hope to spur the adoption of newly proposed contributions, unleashing their potential and concretizing the view of @xcite , according to which "machine learning (ml) engineers should enrich their evaluation toolkits with more flexible, semantic-level metrics". ease-of-use the focus on simplicity is another key factor in fostering impact and usability, allowing users to write less code, reduce errors, and prototype faster. it is also meant to minimize the implementational burden and quickly move from papers to practical applications. we concentrate our efforts on designing an intuitive application programming interface (api) accompanied by rich documentation with a curated list of executable notebooks and examples. this makes the software useful for both academia and industry. reproducibility reproducibility is a core concept of utmost concern in ml and nlp, a prerequisite to trustworthiness. nlg evaluation exacerbates the problem even more, with well-known plagues like heavy undocumented preprocessing pipelines, non-transparent dataset selections, and concealed parameter settings @xcite @xcite . a critical design objective of nlg-metricverse is permitting experimental evaluation results to be seamlessly reproduced, promoting a fully detailed specification. in this way, users can simply integrate their original research into the shared codebase and fairly compare their solution with the existing literature. besides serving for sound and consistent scientific research, reproducibility is a means to speed up the development of new metrics. when it comes to modelbased metrics, transparency also applies to hardware setup, runtime measures, and co2 impact. ## framework nlg-metricverse is implemented as a python library that provides a wrapper around a panoply of nlg evaluation metrics and complementary needs. regardless of the task, an nlg model generally produces one or more predictions (i.e., hypotheses, candidates) @xmath0, . . . , p k conditioned on a given context or source @xmath1, . . . , c p . then, one or multiple human-created references (i.e., groundtruths) @xmath2, . . . , r l may be provided to assist the evaluation. in ## main modules nlg-metricverse is organized into three main modules: metrics ( §5.1), meta-evaluation ( §5.2), and visualization ( §5.3). the library is intended to be a continuous and collaborative project, extended as new solutions become available. in what follows, we describe the features provided at the current stage of development. figure 1 shows the operational representation of the modules and their interplay within the framework detailed in §4. nlg-metricverse is in turn built on top of opensource libraries, including datasets @xcite , numpy (van der walt et al., 2011), scipy @xcite , and matplotlib @xcite . where possible, metrics are implemented using canonical repositories released by authors. ## case study: graph-augmented biomedical abstractive summarization in this section, we use nlg-metricverse to examine the summaries generated by a language model infused with semantic parsing graphs. injecting explicit semantic structures-like events @xcite , abstract meaning representations (amrs) @xcite , and corpus-level knowledge @xcite )-is a new trend followed by the nlp community to overcome lexical superficiality and draw a complementary path to architectural scaling, fundamental in low-resource settings @xcite . graph-augmented methods unlock a higher level of abstraction and more accurate emulation of human interpretation, rewriting, and paraphrasing. faced with semantic-driven models, researchers must avoid being confined to traditional overlap-based metrics and monolithic quality dimensions, thus outlining a valuable testbed for our library. ## conclusion the nlg evaluation community demands efforts toward making research more transparent, reproducible, and open. easy access to a wide variety of automatic metrics and related features holds a lot of potential. a central hub would democratize research, increase comparability, mitigate the computational/implementational burden, and hopefully steer innovation to more robust contributions. in fact, researchers would be able to evaluate their nlg systems at scale without being limited to very few metrics whose code is easily available. they would also be able to critically examine existing metrics, perform white-box attacks, or carefully craft adversarial examples. with nlg-metricverse, we take an important step towards a single, unified, coherent, end-toend, and easily extendable framework for nlg evaluation. a solid reference point and shared resource for researchers and practitioners working in the area. being a community-driven effort, we plan in both the near and medium terms to support more recent task-specific metrics, benchmarks, meta-evaluation techniques for robustness, and skew factor analyses. we also intend to include more document-level measures. we hope that this library may trigger a positive reinforcement loop within our community, nudging it to explore the metric universe.