AMR Parsing is Far from Solved: G r APES , the Granular AMR Parsing Evaluation Suite (2023)

Abstract: We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge set for Abstract Meaning Representation (AMR) parsing with accompanying evaluation metrics. AMR parsers now obtain high scores on the standard AMR evaluation metric Smatch, close to or even above reported inter-annotator agreement. But that does not mean that AMR parsing is solved; in fact, human evaluation in previous work indicates that current parsers still quite frequently make errors on node labels or graph structure that substantially distort sentence meaning. Here, we provide an evaluation suite that tests AMR parsers on a range of phenomena of practical, technical, and linguistic interest. Our 36 categories range from seen and unseen labels, to structural generalization, to coreference. GrAPES reveals in depth the abilities and shortcomings of current AMR parsers.

## introduction abstract meaning representation (amr; @xcite ) parsing, the task of predicting a graph like the one in fig. 1 for a given sentence, has improved by leaps and bounds in recent years. in fact, parsing performance of recent parsers, as evaluated by smatch score @xcite , has reached and even surpassed reported human interannotator agreement @xcite . has amr parsing reached human performance, and thus has this form of semantic parsing been solved? @xcite perform human-expert evaluation for amr parsing and find that, no, amr parsing is far from solved: only 62% of parser outputs were rated acceptable. in this work, we present an evaluation suite for english amr parsing, including new data end metrics, that measures performance of amr parsers with unprecedented breadth, detail and accuracy. instead of computing a single score, like smatch, on a single test set, we evaluate parsing performance on a range of phenomena of practical, technical and linguistic interest. we answer the questions of how well parsers can handle pragmatic coreference, ellipsis, pp attachment, rare words, unseen named entities and structural generalization, just to name a few. our granular amr parsing evaluation suite (grapes) combines a selection of existing and novel sentence-amr pairs. a central theme in our development of grapes was that our metrics should actually evaluate what they promise to evaluate. to this end, we developed novel evaluation metrics specifically designed for the evaluation categories in our dataset. we also annotated, where necessary by hand, which graphsentence pairs are correct and unambiguous, as well as which pairs are relevant for each category. we further annotated what part of each graph is relevant, reducing possible distractors. our work has three goals. first, to give quantitative results on a set of different phenomena in amr parsing, so that we as a community know what abilities amr parsers have, and where we can trust them to get it right. second, to provide a tool that compares amr parsers in more detail, and makes their differences visible. and third, to allow developers of amr parsers to see where their systems still struggle -what needs to be improved. in this sense, grapes also functions as a challenge to the community. the experimental results we present in this paper confirm that grapes can serve these three purposes. our main contributions are: grapes is available open source at https:// github.com/jgroschwitz/grapes . we start by discussing related work in section 2. we describe our categories and why we selected them in section 3, and our measures to achieve high quality data and metrics in section 4. we follow this up with an analysis of results for three recent amr parsers (section 5) and recommendations for how to use our evaluation suite (section 6). ## related work evaluation for amr parsing is currently an active field of research. the standard evaluation metric to compare monolingual amrs (both parsed and human annotated) is smatch @xcite , which uses a graph matching algorithm to compute precision, recall, and f-score for semantic overlap of nodes and edges. though designed to measure overall semantic adequacy of a predicted graph, recent advances in amr parsing performance appear to have outgrown smatch, spawning research on how to develop more fine-grained and interpretable evaluation metrics. @xcite , in addition to computing traditional smatch on parser results, present variants of smatch that emphasize 'core semantics' based on triple relations to the root, or predicative core, of the amr graph. similarly, @xcite show that smatch often misses small but significant semantic divergences and recommend supplementing the metric with other metrics (including human analysis) to measure more fine-grained semantic adequacy. @xcite extends this work and separates smatch into scores for pre-processing, alignment, and scoring. the above works reveal inconsistencies in how smatch measures parser performance and point to the need for more focused evaluation methods. targeted evaluations for amr parsing in the form of phenomenon-specific benchmarks aim to address this need. @xcite investigate the role of reentrant edges specifically. @xcite is the closest analogue of grapes, presenting an evaluation metric that pinpoints distinct subtasks of amr parsing. they introduce a set of nine metrics that measure challenging lin-guistic phenomena such as reentrancies and named entities. we take this idea further, evaluating on 36 categories, expanding both breadth and depth. moreover, we put additional emphasis on disentangling parsing performance on specific phenomena from overall performance, resulting in more precise and interpretable metrics. finally, grapes includes newly annotated amr-sentence pairs (both hand-built and grammar-generated) specifically designed to evaluate certain phenomena. beyond amr evaluation, a growing literature on structural generalization is relevant to the amr parsing task in developing evaluation suites for difficult semantic phenomena rooted in linguistic structure. for example, in the cogs dataset @xcite parsers must predict logical forms for sentences comprised of novel structural combinations, mimicking human ability to generalize compositionally. this dataset has recently been extended in the slog dataset for semantic parsing @xcite , which targets more difficult structural phenomena using the same logical form. grapes takes both amr and non-amr benchmarks as inspiration for a more comprehensive evaluation for amr. we include several existing tasks into our dataset: (i) the winograd schema challenge @xcite , which consists of pairs of sentences with pronominal coreference that require pragmatics to disambiguate; (ii) putting words into bert's mouth @xcite for word disambiguation on simple sentences; and (iii) the unbounded dependencies corpus @xcite which contains real life examples of long range dependencies in different categories. ## challenge categories we cover a broad range of phenomena that are interesting from practical, technical, and linguistic perspectives. ## dataset and evaluation design with a wide range of phenomena selected, our guiding principles in creating the actual dataset and the evaluation metrics that form grapes are: ## recommendations for researchers in amr parsing who want to show their parser's results on grapes, we recommend including the more compact ## conclusion we have shown that state-of-the-art amr parsers still struggle with various phenomena, including data sparsity, contextual ambiguity resolution and structural generalization. we provide a detailed evaluation suite with custom metrics to measure progress in these areas.