Weisfeiler-Leman in the Bamboo: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity (2021)

Abstract: Several metrics have been proposed for assessing the similarity of (abstract) meaning representations (AMRs), but little is known about how they relate to human similarity ratings. Moreover, the current metrics have complementary strengths and weaknesses: Some emphasize speed, while others make the alignment of graph structures explicit, at the price of a costly alignment step. In this work we propose new Weisfeiler-Leman AMR similarity metrics that unify the strengths of previous metrics, while mitigating their weaknesses. Specifically, our new metrics are able to match contextualized substructures and induce n:m alignments between their nodes. Furthermore, we introduce a Benchmark for AMR Metrics based on Overt Objectives (Bamboo), the first benchmark to support empirical assessment of graph-based MR similarity metrics. Bamboo maximizes the interpretability of results by defining multiple overt objectives that range from sentence similarity objectives to stress tests that probe a metric’s robustness against meaning-altering and meaning- preserving graph transformations. We show the benefits of Bamboo by profiling previous metrics and our own metrics. Results indicate that our novel metrics may serve as a strong baseline for future work.

## introduction meaning representations aim at capturing the meaning of text in an explicit graph format. a prominent framework is abstract meaning representation (amr), proposed by @xcite . amr views sentences as rooted, directed, acyclic, labeled graphs. their nodes are variables, attributes, or (open-class) concepts and are connected with edges that express semantic relations. there are many use cases in which we need to compare or relate two amr graphs. a common situation is found in parser evaluation, where amr metrics are widely applied @xcite . 1 yet, there are more situations where we need to measure similarity of meaning as expressed in amr graphs. for example, @xcite leverage amr metrics in a semantic search engine for covid-19 queries, @xcite use metric feedback to reinforce amr parsers, @xcite emulates metrics for referenceless amr ranking and rating, and @xcite use amr metrics for nlg evaluation. so far, multiple amr metrics @xcite @xcite @xcite have been proposed to assess amr similarity. however, due to a lack of an appropriate evaluation benchmark, we have no empirical evidence that could tell us more about their strengths and weaknesses or offer insight about which metrics may be preferable over others in specific use cases. additionally, we would like to move beyond the aforementioned metrics and develop new metrics that account for graded similarity of graph substructures, which is not an easy task. however, it is crucial when we need to compare amr graphs in a deeper way. consider figure 1 , which shows two amrs that convey very similar meanings. all aforementioned metrics assign this pair a low similarity score, and-if alignment-based, as is smatch @xcite )-find only subpar alignments. 2 in this case, we want a metric that provides us with a high similarity score and, ideally, an explanatory alignment. the structure of this paper is as follows. in §2 we discuss related work. in §3 we describe our first contribution: new amr metrics that aim at unifying the strengths of previous metrics while mitigating their weaknesses. specifically, our new metrics are capable of matching larger substructures and provide valuable n:m alignments in polynomial time. in §4 we introduce bamboo , our second contribution: it is the first benchmark data set for amr metrics and includes novel robustness objectives that probe the behavior of amr metrics under meaning-preserving and meaning-altering transformations of the inputs ( §5). in §6 we use bamboo for a detailed, multi-faceted empirical study of previous and our proposed amr metrics. we release bamboo and our new metrics. 3 2 related work the classical amr metric and its adaptions the 'canonical' and widely applied amr metric is smatch (semantic match) @xcite . it solves an np-hard graph alignment problem approximately with a hill-climber and scores matching triples. smatch has been adapted to s 2 match (soft semantic match), by @xcite to account for graded similarity of concept nodes (e.g., cat-kitten), using word embeddings. smatch has also been adapted by @xcite in w(eighted)smatch (wsmatch), which penalizes errors relative to their distance to the root. this is motivated by the hypothesis that ''core semantics'' tend to be located near a graph's root. ## bamboo : creating the first benchmark for amr similarity metrics we now describe the creation of bamboo , which aims to provide the first benchmark that allows researchers to empirically (i) assess amr metrics, (ii) compare amr metrics, and possibly (iii) train amr metrics. grounding amr similarity metrics in human ratings of semantic sentence similarity as the main criterion for assessing amr similarity metrics, we use human judgments of the meaning similarity of sentences underlying pairs of amrs. a corresponding principle has been proposed by opitz et al. (2020): a metric of pairs of amr graphs g and g that represent sentences s and s should reflect human judgments of semantic sentence similarity and relatedness: amrm etric(g, g ) ≈ humanscore(s, s ) (11) similarity objectives accordingly, we select, as evaluation targets for amr metrics, three notions of sentence similarity, which have previously been operationalized in terms of human-rated evaluation datasets: (i) the semantic textual similarity (sts) objective from baudiš et al. (2016a,b); (ii) the sentence relatedness objective (sick) from marelli et al. (2014); (iii) the paraphrase detection objective (para) by @xcite . each of these three evaluation data sets can be seen as a set of pairs of sentences (s i , s i ) with an associated score humanscore(•) that provides the human sentence relation assessment score reflecting semantic similarity (sts), semantic relatedness (sick) and whether sentences are paraphrastic (para). hence, each of these data sets can be described as {(s i , s i , humanscore (s i , s i ) = y i )} n @xmath0. both sts and sick offer scores on likert scales, ranging from equivalence (max) to unrelated (min), while para scores are binary, judging sentence pairs as being paraphrases (1), or not (0). we min-max normalize the likert scale scores to the range [0, 1] to facilitate standardized evaluation. for bamboo , we replace each pair (s i , s i ) with their amr parses: (p @xmath1, p @xmath2, transforming the data into {(p i , p i , y i )} n @xmath3. this provides the main partition of the graph statistics data instances (s. length) # nodes density source train/dev/test avg. 50 th avg. 50 th avg. 50 th sts 5749/1500/1379 9.9 8 14.1 12 0.10 0.08 sick 4500/500/4927 9.6 9 10.7 10 0.11 0.1 para 3576/500/1275 18.9 19 30.6 30 0.04 0. 04 manual data quality assessment: three-way graph quality ratings from each data set (sick, sts, para) we randomly select 100 sentences and create their parses with t5s2s. additionally, to establish a baseline, we also parse the same sentences with the gpla parser of @xcite , a neural graph prediction system that uses latent alignments (which reports 74.4 smatch score on amr2). this results in 300 gpla parses and 300 t5s2s parses. a human annotator 9 inspects the (shuffled) sample and assigns three-way labels: flawed-an amr contains critical errors that distort the meaning significantly; silver-an amr contains small errors that can potentially be neglected; gold-an amr is acceptable. results in specifically, we propose three transformation strategies. (i) reification (reify ), which changes 9 the human annotator is a proficient english speaker and has worked several years with amr. ## experimental insights questions posed to bamboo bamboo allows us to address several open questions: the first set of questions aims to gain more knowledge about previously released metrics. that is, we would like to know: what semantic aspects of amr does a metric measure? if a metric has hyper-parameters (e.g., sembleu), which hyper-parameters are suitable (for a specific objective)? does the costly alignment of smatch pay off, by yielding better predictions, or do the faster alignment-free metrics offer a 'free-lunch'? a second set of questions aims to evaluate our proposed novel amr similarity metrics, and to assess their potential advantages. ## conclusion our contributions in this work are three-fold: (i) we propose a suite of novel weisfeiler-leman amr similarity metrics that are able to reconcile a performance conflict between precision of amr similarity ratings and the efficiency of computing alignments. (ii) we release bamboo , the first benchmark that allows researchers to assess amr metrics empirically, setting the stage for future work on graph-based meaning representation metrics. (iii) we showcase the utility of bamboo , by applying it to profile existing amr metrics, uncovering hitherto unknown strengths or weaknesses, and to assess the strengths of our newly proposed metrics that we derive and further develop from the classic weisfeiler-leman kernel. we show that through bamboo we are able to gain novel insight regarding suitable hyperparameters of different metric types, and to gain novel perspectives on how to further improve amr similarity metrics to achieve better correlation with the degree of meaning similarity of paired sentences, as perceived by humans.