A Dynamic, Interpreted C heck L ist for Meaning-oriented NLG Metric Evaluation – through the Lens of Semantic Similarity Rating (2022)

Abstract: Evaluating the quality of generated text is difficult, since traditional NLG evaluation metrics, focusing more on surface form than meaning, often fail to assign appropriate scores. This is especially problematic for AMR-to-text evaluation, given the abstract nature of AMR.Our work aims to support the development and improvement of NLG evaluation metrics that focus on meaning by developing a dynamic CheckList for NLG metrics that is interpreted by being organized around meaning-relevant linguistic phenomena. Each test instance consists of a pair of sentences with their AMR graphs and a human-produced textual semantic similarity or relatedness score. Our CheckList facilitates comparative evaluation of metrics and reveals strengths and weaknesses of novel and traditional metrics. We demonstrate the usefulness of CheckList by designing a new metric GraCo that computes lexical cohesion graphs over AMR concepts. Our analysis suggests that GraCo presents an interesting NLG metric worth future investigation and that meaning-oriented NLG metrics can profit from graph-based metric components using AMR.

## introduction abstract meaning representation @xcite ) has become popular in nlp, one of the reasons being that amr captures the essence of a sentence's meaning, while abstracting away from syntactic idiosyncrasies. especially amr-to-text generation @xcite @xcite has received much attention for applications that require text generation from structured content. however, the evaluation of text generated from amr has been argued to be unsatisfactory @xcite . also, @xcite show that the syntactic diversity of sentences generated from amr is challenging for traditional nlg metrics, especially when candidates differ from the reference in surface properties. several metrics have been proposed that aim to rate the similarity of the meaning of sentences or phrases @xcite ; @xcite ; @xcite ). however, it is difficult to judge where exactly such a metric fails, making it hard for developers to further improve it. to address similar problems, ribeiro et al. (2020) recently proposed a "task-agnostic methodology for testing nlp models" called checklist. they argue that such a method should be used for testing nlp systems instead of solely relying on automatic metrics, which can overestimate a model's performance. similar processes have been applied in early nlp research, e.g. with the tsnlp testsuite @xcite . inspired by checklist, in this work we aim to build a testsuite to enable systematic study and development of nlg evaluation metrics, with a focus on meaning. given the high variability of surface realizations that can be mapped into a single amr graph, building reliable amr-to-text nlg evaluation metrics is hard. hence, it can be useful to construct a systematic checklist, organized around diverse linguistic properties, to measure the performance of different metrics in an interpretable way. we frame our proposed checklist 1 and analyses derived from it in an amr-to-text nlg setting, and focus especially on a metric's capability to assess how well a specific meaning component of an amr is reflected in its textual realization. we measure this using sentence pairs that differ in single linguistic aspects and measure how well various nlg metrics are able to rate such meaning differences. we compare the metric scores to human judgments from semantic textual similarity (sts) and relatedness datasets and analyze the metrics using our interpreted checklist (an outline is shown in fig. 1 ). our contributions in this work are as follows: ## semantic phenomena we consider structural and lexical phenomena that are likely to affect a sentence's meaning. details and example amrs are given in appendix a.4. foot_3 ## conclusion we introduced an extensible checklist for meaning-oriented nlg metrics that allows for comparison of a wide range of nlg metrics. it is interpreted by way of offering test cases grouped by linguistic phenomena. our analyses showcase how checklist can be used to compare metrics, to reveal their strengths and weaknesses. they align with a number of hypotheses, but also show surprising effects, opening avenues to further improve nlg evaluation metrics. we propose a novel, hybrid similarity metric graco that builds cohesion graphs over contextualized amr concept nodes. the metric can focus on contrastive subgraphs, which yields strong correlation with human judgements for negation. with regard to current practice in amr-to-text evaluation, we find evidence that meaning-oriented graph-based metrics present advantages over typical text-based metrics, confirming the findings of @xcite ; @xcite . therefore we recommend to include graph metrics or hybrid graph-and textual metrics into amr-to-text evaluation protocols. our data and code will be publicly available. 11 we welcome contributions to grow it.