Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2021)

Abstract: Systems that generate natural language text from abstract meaning representations such as AMR are typically evaluated using automatic surface matching metrics that compare the generated texts to reference texts from which the input meaning representations were constructed. We show that besides well-known issues from which such metrics suffer, an additional problem arises when applying these metrics for AMR-to-text evaluation, since an abstract meaning representation allows for numerous surface realizations. In this work we aim to alleviate these issues by proposing \mathcal{M}\mathcal{F}_\beta , a decomposable metric that builds on two pillars. The first is the principle of meaning preservation : it measures to what extent a given AMR can be reconstructed from the generated sentence using SOTA AMR parsers and applying (fine-grained) AMR evaluation metrics to measure the distance between the original and the reconstructed AMR. The second pillar builds on a principle of (grammatical) form that measures the linguistic quality of the generated text, which we implement using SOTA language models. In two extensive pilot studies we show that fulfillment of both principles offers benefits for AMR-to-text evaluation, including explainability of scores. Since \mathcal{M}\mathcal{F}_\beta does not necessarily rely on gold AMRs, it may extend to other text generation tasks.

## introduction abstract meaning representation @xcite ) aims at capturing the meaning of a sentence in a machine-readable graph format. amr captures, i.a., word senses, semantic roles and coreference. the amr in fig. 1 represents the sentence perhaps, the parrot is telling itself a story. in this graph, tell-01 links to a propbank @xcite pant roles: parrot is both speaker (arg0) and hearer (arg2), story is the utterance (arg1). the task of amr-to-text generation has recently garnered much attention @xcite @xcite @xcite . the output of amr-to-text systems is typically evaluated against the sentence from which the amr was created, using standard surface string matching metrics such as bleu @xcite or chrf(++) @xcite @xcite , as is standard in many nlg tasks. these metrics suffer from several issues, for example, they penalize paraphrases, are highly sensitive to outliers @xcite , and lack interpretability @xcite . some of these issues get compounded when evaluating amr-to-text. the core of the problem is that there are many ways to realize a sentence from a meaning representation. fig. 2 shows four candidate sentences (i-iv) for a given amr (left). one system generates (i): maybe the cat is playing. while another generates (iii): perhaps, the cat plays the flute. clearly, (i) captures the meaning of the gold graph better than (iii), which contains 'hallucinated' content -a well-known issue in neural generation @xcite ). yet, when using a canonical metric such as bleu to evaluate sentences (i) and (iii) against the reference, the system that produces hallucinations (iii) is greatly rewarded (54 bleu points) to the disadvantage of systems that yield meaning preserving possible-01 play-01 cat generated candidates: i: maybe the cat is playing. iv: mayybe the cat are playing. ## related work traditionally, the performance of nlg systems has been evaluated with word n-gram matching metrics such as the popular bleu metric in mt @xcite or rouge @xcite in document summarization. yet, such metrics suffer from several well-known issues @xcite @xcite . e.g., due to their symbolic matching strategy they cannot account for paraphrases. recently, unsupervised @xcite or learned metrics @xcite based on contextual language models have been proposed. for example, bertscore @xcite uses bert @xcite to encode candidate and reference and computes a score based on a cross-sentence word-similarity alignment. compared with bleu, it is computationally more expensive but tends to show higher agreement with humans. however, all of the aforementioned metrics return scores that are hardly interpretable and we cannot tell what exactly they have measured. these problems carry over to the evaluation of amr-to-text generation: @xcite find that bleu does not well correspond to human ratings of generations from amr, and @xcite show through human analysis that none of the existing automatic metrics can provide nuanced views on generation quality. our proposal mf β takes a first step to address these issues by aiming at a clear separation of form and meaning, as called for by @xcite . first attempts of assessing semantic generation quality have been examined in mt using semantic role labeling @xcite or wsd and nli @xcite , in-between lies spice that evaluates caption generation via inferred semantic propositions @xcite . just like mf β , spice relies on automatic parses (a dependency parse of the caption and a scene graph predicted for the image) to evaluate content overlap of image and caption. thus, spice is a direct precursor of an nlg metric in v&l that relies on automatically produced structured representations. our work extends this previous work by showing ways of probing potentially harmful effects of incorporating automatic parsing components. ## conclusion we propose mf β score, a new metric for evaluation of text generation from (abstract) meaning representation. the metric is built on two pillars: f orm measures grammaticality and fluency of the produced sentences and m eaning assesses to what extent the meaning of the input amr is reflected in the produced sentence. we show that mf β has the potential to yield fine-grained performance assessment that go beyond what conventional metrics can provide. using its β-parameter, mf β can be decomposed into complementary views -m eaning and f orm -paving the way for custom gauging and selection of nlg systems. we have seen that mf β corresponds well to bertscore when rankings systems, but overcomes its opaqueness by disentangling m eaningand f orm-related quality aspects. in sharp contrast to bertscore, the f orm component of mf β dispenses with string matching against reference sentences, offering an assessment independent of lexical alignment. an important hyperparameter of our metric is the required amr parsing component for meaning reconstruction. we investigate the impact of its choice by choosing alternative high-performing parsers. our study shows that absolute metric scores tend to increment when using a better parser, while system rankings are quite stable. furthermore, we outline the potential of mf β to extend to further text generation tasks, by ablating the human gold graph from the evaluation, such that the metric score can be computed from candidate and reference text alone. since benchmarking of systems needs deeper exploration, we recommend mf β score to obtain better diagnostics and explainability of text generation systems, including, but not limited to (a)mr-to-text.