A Human Evaluation of AMR -to- E nglish Generation Systems (2020)

Abstract: Most current state-of-the art systems for generating English text from Abstract Meaning Representation (AMR) have been evaluated only using automated metrics, such as BLEU, which are known to be problematic for natural language generation. In this work, we present the results of a new human evaluation which collects fluency and adequacy scores, as well as categorization of error types, for several recent AMR generation systems. We discuss the relative quality of these systems and how our results compare to those of automatic metrics, finding that while the metrics are mostly successful in ranking systems overall, collecting human judgments allows for more nuanced comparisons. We also analyze common errors made by these systems.

## introduction abstract meaning representation, or amr @xcite , is a representation of the meaning of a sentence as a rooted, labeled, directed acyclic graph. for example, (l / label-01 :arg0 (c / country :wiki "georgia _ (country)" :name (n / name :op1 "georgia")) :arg1 (s / support-01 :arg0 (c2 / country :wiki "russia" :name (n2 / name :op1 "russia"))) amr generation is the task of generating a sentence in natural language (in this case, english) from an amr graph. this has applications to a range of nlp tasks, including summarization @xcite and machine translation @xcite . like other natural language generation (nlg) tasks, this is difficult to evaluate due to the range of possible valid sentences corresponding to any single amr. currently, amr generation systems are often evaluated only with automatic metrics that compare a generated sentence to a single human-authored reference; for amr, this is the sentence from which the amr graph was created. however, there is evidence that these metrics may not be a good representation of human judgments for amr generation @xcite and nlg in general (see §2.1). thus, in this work, we present a new human evaluation of several recent amr generation systems, most of which had not previously been manually evaluated. our methodology ( §3) differs in several ways from previous evaluations of amr generation, including separate direct assessment of fluency and adequacy; and asking annotators to evaluate sentences without comparison to a reference, in order to avoid biasing them toward a particular wording. we analyze ( §4) what our results show about the relative quality of the systems and how this compares to their scores from automatic metrics, finding that these metrics are mostly accurate in ranking systems, but that collecting separate judgments for fluency, adequacy, and error types allows us to characterize the relative strengths and weaknesses of each system in more detail. finally, we discuss common errors among sentences which received low scores from annotators, identifying issues for future researchers to address including hallucination, anonymization, and repetition. ## background in §2.1 we discuss previous work on evaluation of amr generation and related nlg tasks, both with automatic metrics and human evaluation. in §2.2 we survey recent work in amr generation, including describing the systems which we evaluate. ## methodology we conduct a human evaluation of several amr generation systems. §3.1 discusses the general survey design, while §3.2 discusses details of the pilot survey, which validates the methodology by applying it to data from the semeval evaluation, and §3.3 discusses the evaluation of more recent systems. ## conclusion and future work our analysis of these systems, and especially of their common errors, points toward directions for researchers developing nlg systems, especially for amr, to improve their output. we recommend attempting to find solutions to the common issues that led to low scores even from state-of-the-art systems, such as anonymization of infrequent concepts, unnecessary repetition of words, and hallucination. while this study found that popular automatic metrics were mostly successful in ranking these systems in the same order human annotators did, we also found that the human evaluation was able to identify strengths and weaknesses of systems with more nuance than a single number can convey. we also acknowledge that, given prior work pointing to the inadequacy of metrics such as bleu for nlg and amr generation, more research is needed to determine the reliability of these metrics for comparing systems. we suggest that researchers in amr generation and other nlg tasks continue to supplement automatic metrics with human evaluation as much as possible.