Referenceless Parsing-Based Evaluation of AMR -to- E nglish Generation (2021)

Abstract: Reference-based automatic evaluation metrics are notoriously limited for NLG due to their inability to fully capture the range of possible outputs. We examine a referenceless alternative: evaluating the adequacy of English sentences generated from Abstract Meaning Representation (AMR) graphs by parsing into AMR and comparing the parse directly to the input. We find that the errors introduced by automatic AMR parsing substantially limit the effectiveness of this approach, but a manual editing study indicates that as parsing improves, parsing-based evaluation has the potential to outperform most reference-based metrics.

## introduction natural language generation (nlg) is notoriously difficult to evaluate well due to its one-to-many nature: thanks to the infinite capacity of human language, any given meaning can be expressed in a potentially unlimited number of ways. thus, listing the 'right answer(s)' and comparing a system's output against such a list, which is a possible evaluation method for many other tasks, is fundamentally limited for nlg. nevertheless, automatic evaluation of nlg has traditionally been dominated by reference-based metrics like bleu @xcite . in recent years, however, referenceless evaluation metrics have been gaining popularity in nlg and related fields. in this paper we examine the potential and limitations of one such approach: using semantic parsing to compare a generated sentence to a meaning representation from which it was generated, in order to measure semantic adequacy. we focus on generation of english text from abstract meaning representation graphs ("amrs"; @xcite . figure 1 shows an example of an amr, which represents the meaning of a sentence. amr does not represent certain morphological and syntactic details such as tense, number, definiteness, and word order, so the graph shown could represent a number of alternate sentences, such as: we examine the hypothesis that we can measure the semantic adequacy of a sentence generated from an amr by performing the reverse operation-namely, parsing the generated sentence into amr-and measuring the similarity of the parsed amr graph to the original. in essence, this idea exploits complementarity of english-to-amr parsers and amr-to-english generators being evaluated. assuming an accurate parse, we would expect this to be a good measure of the adequacy of the generated sentence, since a sentence that accurately expresses the meaning in the original amr should have the same amr. we further formalize this approach in §2. as discussed in §3, this method has also been suggested by @xcite ; we contribute new analyses of its validity, in particular by measuring its correlation with human adequacy judgments collected by @xcite . we find that errors made by an automatic amr parser substantially limit the quality of parsing-based evaluation as a proxy for human evaluation, resulting in a lower correlation with adequacy scores than many reference-based metrics ( §5). to approximate an upper bound for the potential of this evaluation approach with improved parsing, we conduct an additional study using manually-corrected amr parses; we find that this substantially improves the quality of the metric ( §6). ## parsing-based evaluation amr-to-english generation is the task of taking an input amr graph a and generating a sentence g expressing the meaning content of the amr in english. ideally, we would evaluate generation by comparing g directly to a to determine how well g expresses the meaning in a; this is what the human annotators whose judgments we use (see §4) did. however, we don't know of an existing way to directly compare a sentence to an amr graph; instead, our metrics tend to compare two items of the same type. reference-based metrics compare the generated sentence g to r, the english sentence for which the amr was created, and which is typically used as the sole reference in evaluation. we analyze the hypothesis that we can more accurately capture the details relevant to amr by comparing amrs to each other: specifically, comparing a to either p, an automatic parse of g, or to p ′ , a manually-corrected version of p. our notation is summarized in ## background the evaluation method we analyze in this paper is closely related to the mf b metric suggested by @xcite , which combines a measure of meaning preservation, m, with a language model-based measure of grammatical form, f. their meaning preservation metric, m, assigns a score to a generated sentence by parsing it into amr and computing the parse's similarity to the gold amr. they use the amr parser by @xcite and the s 2 match similarity metric; we experiment with these as well as other options for both parser and metric. while they perform a number of pilot experiments to test the robustness of mf b , such as its performance with different parsers, opitz and frank do not test the correlation of their metric with human judgments; thus, the work presented here adds to our understanding of the validity of this type of metric as a proxy for human evaluation. as a baseline, we also compare the results of parsing-based evaluation with several referencebased metrics, including those that have traditionally been used to evaluate amr generation as well as newer metrics that have shown promising results for nlg. ## data we use human judgment data from @xcite , consisting of judgments on a total of 600 sentences: 100 human-authored reference sentences and their corresponding amrs, and 500 sentences automatically generated from these amrs by 5 different systems. when comparing to reference-based automatic metrics, we do not score the reference sentences themselves since, with only one reference per amr, these would trivially receive perfect scores on such metrics. each sentence has a score on a scale of 0-100, for each of fluency and adequacy, averaged over two annotators. we compare automatic metrics to these judgments, and particularly to the adequacy judgments, since we are primarily interested in parsing-based evaluation as a proxy for adequacy evaluation. annotators additionally provided binary judgments on whether information was added or omitted, and whether the sentence was incomprehensible; we use the latter of these judgments in §6 to determine which generated sentences to manually edit the parses of. ## experiment 1: automatic metrics this section describes experiments with variations on the automatic version of the parsing based metric; that is, the use of similarity metrics comparing the automatic parse p to the gold amr a. we experiment with different amr parsers ( §5.1) and variations on the smatch similarity metric ( §5.2) and measure the correlation to human judgments of adequacy ( §5.3). ## experiment 2: manually-edited parses even a state-of-the-art amr parser is of course not perfect, and may struggle more with parsing automatically-generated sentences than the humanauthored ones it is designed for. the potential for parser error is a major limitation of the proposed approach; evaluating the parse p against gold amr a can only be a good measure of g's relationship to a if p is a sufficiently accurate parse of g. thus, to get a better sense of the effect that parsing errors can have on this metric even when using a sota parser, and of a rough upper bound for how well the metric could work in the future as parsing improves, we also manually edited a sample of the parses p to create alternate parses, p ′ , which better reflect the meaning expressed in the generated sentences g, and use smatch to compare p ′ to a. ## analysis a common parser error was failure to recognize named entities when they were not capitalized; examples of this are given in tables 7 and 8. as figure 2 shows, three of the systems never produce capitals in their output, while those of konstas and manning typically produce about as many capitals as are present in the reference. thus, it seems likely that the systems that never produce capitals may be unfairly penalized by a parsing-based metric. it may be possible to overcome this particular limitation of the automatic parser by adding a preprocessing step that recognizes and capitalizes named entities, or by training the parser on more all-lowercase examples. ## conclusion in this paper, we have explored the idea of evaluating amr generation via amr parsing and similarity metrics, using the human judgments of adequacy collected by @xcite to test the validity of possible variants of the parsingbased metric approach and compare them to existing reference-based metrics. we found that parser quality is a major factor affecting the performance of this evaluation approach: the better the amr parser, the better the evaluation; however, even a state-of-the-art parser with an accuracy of 80+% on standard human-authored data has significant limitations for evaluating generated sentences, including a failure to recognize named entities in the absence of capitalization. we showed that when automatic amr parses are manually edited to better reflect the meaning in generated sentences, this referenceless metric outperforms most popular automatic reference-based metrics, including bleu and bertscore (but not bleurt). while the current reliance on manual editing for more reliable results may not be practical for evaluation, the results of this experiment indicate that fully-automatic parser-based metrics are likely to prove more reliable in the future as the state of the art in amr parsing continues to improve, especially if newer amr generation systems also more closely replicate human-authored data, such as by producing more human-like capitalization than the majority of systems tested here did.