AMR Similarity Metrics from Principles (2020)

Abstract: Different metrics have been proposed to compare Abstract Meaning Representation (AMR) graphs. The canonical Smatch metric (Cai and Knight, 2013) aligns the variables of two graphs and assesses triple matches. The recent SemBleu metric (Song and Gildea, 2019) is based on the machine-translation metric Bleu (Papineni et al., 2002) and increases computational efficiency by ablating the variable-alignment. In this paper, i) we establish criteria that enable researchers to perform a principled assessment of metrics comparing meaning representations like AMR; ii) we undertake a thorough analysis of Smatch and SemBleu where we show that the latter exhibits some undesirable properties. For example, it does not conform to the identity of indiscernibles rule and introduces biases that are hard to control; and iii) we propose a novel metric S2 match that is more benevolent to only very slight meaning deviations and targets the fulfilment of all established criteria. We assess its suitability and show its advantages over Smatch and SemBleu.

## introduction proposed in 2013, the aim of abstract meaning representation (amr) is to represent a sentence's meaning in a machine-readable graph format (banarescu et al., 2013). amr graphs are rooted, acyclic, directed, and edge-labeled. entities, events, properties, and states are represented as variables that are linked to corresponding concepts (encoded as leaf nodes) via is-instance relations (cf. figure 1 , left). this structure allows us to capture complex linguistic phenomena such as coreference, semantic roles, or polarity. when measuring the similarity between two amr graphs a and b, for instance for the purpose of amr parse quality evaluation, the metric of choice is usually smatch (cai and knight, 2013). its backbone is an alignment-search be-tween the graphs' variables. recently, the sembleu metric (song and gildea, 2019) has been proposed that operates on the basis of a variable-free amr (figure 1 , right), foot_0 converting it to a bag of k-grams. circumventing a variable alignment search reduces computational cost and ensures full determinacy. also, grounding the metric in bleu (papineni et al., 2002) has a certain appeal, since bleu is quite popular in machine translation. however, we find that we are lacking a principled in-depth comparison of the properties of different amr metrics that would help informing researchers to answer questions such as: which metric should i use to assess the similarity of two amr graphs, e.g., in amr parser evaluation? what are the trade-offs when choosing one metric over the other? besides providing criteria for such a principled comparison, we discuss a property that none of the existing amr metrics currently satisfies: they do not measure graded meaning differences. such differences may emerge because of near-synonyms such as ruin -annihilate; skinny -thin -slim; enemy -foe (inkpen and hirst, 2006; edmonds and hirst, 2002) or paraphrases such as be able to -can; unclear -not clear. in a classical syntactic parsing task, metrics do not need to address this issue because input tokens are typically projected to lexical concepts by lemmatization, hence two graphs for the same sentence tend not to disagree on the concepts projected from the input. this is different in semantic parsing where the projected concepts are often more abstract. this article is structured as follows: we first establish seven principles that one may expect a metric for comparing meaning representations to figure 1 : a cat drinks water. simplified amr graph and underlying deep form with is-instance relations (----á) from variables (solid) to concepts (dashed). satisfy, in order to obtain meaningful and appropriate scores for the given purpose ( §2). based on these principles we provide an in-depth analysis of the properties of the amr metrics smatch and sembleu ( §3). we then develop s foot_2 match, an extension of smatch that abstracts away from a purely symbolic level, allowing for a graded semantic comparison of atomic graph-elements ( §4). by this move, we enable smatch to take into account fine-grained meaning differences. we show that our proposed metric retains valuable benefits of smatch, but at the same time is more benevolent to slight meaning deviations. our code is available online at https://github. com/heidelberg-nlp/amr-metric-suite . ## from principles to amr metrics the problem of comparing amr graphs a, @xmath0, for example, parser evaluation or inter-annotator agreement calculation (iaa). to measure the extent to which a and b agree with each other, we need a metric: d × @xmath1, we use similarity). below we establish seven principles that seem desirable for this metric. ## assessing amr metrics with principles this section evaluates smatch and sembleu against the seven principles we established above by asking: why does a metric satisfy or violate a given principle? and what does this imply? we start with principles from mathematics. ## metrics' effects on parser evaluation we have seen that different metrics can assign different scores to the same pair of graphs. we now want to assess to what extent this affects rankings: does one metric rank a graph higher or lower than the other? and can this affect the ranking of parsers on benchmark datasets? quantitative study: graph rankings in this experiment, we assess whether our metrics rank graphs differently. we use ldc2017t10 (dev) parses by camr [c 1 . . .c n ], jamr [j 1 . . .j n ] and gold graphs [y 1 . . .y n ]. given metrics f and g we obtain results f c := [f(c 1 , y 1 ) . . . f(c n , y n )] and analogously f j , g c and g j . we calculate two figure 10 : an example from sts, where s 2 match yields a score that better reflects the human judgement, due to detecting a similarity between the abstract anaphora it and this . quantitative study: parser rankings having seen that our metrics disagree on the ranking of individual graphs, we now quantify the effects on the ranking of parsers. we collect outputs of three state-of-art parsers on the test set of ldc2017t10: gpla, a sequence-to-graph transducer (stog), and a neural top-down parser (top-down). in sum, by building graphs of higher complexity, gpla takes a greater risk when attaching wrong concepts to connected nodes where errors are penalized more strongly by sembleu than smatch, according to the biases we have studied in §3 ( ## summary of our metric analyses a central insight brought out by our analysis is that sembleu exhibits biases that are hard to control. this is caused by two (interacting) factors: (i) the extraction of k-grams is applied on the graph top to bottom and visits some nodes more frequently than others. (ii) it raises some (but not all) leaf nodes to connected nodes, and these nodes will be overly frequently contained in extracted k-grams. we have shown that these two factors in combination lead to large biases that researchers should be aware of when using sembleu ( §3). its ancestor bleu does not suffer from such biases since it extracts k-grams linearly from a sentence. given that sembleu is built on bleu, it is inherently asymmetric. however, we have shown that the asymmetry (principle iii) measured for bleu in mt is amplified by sembleu in amr, mainly due to the biases it incurs (principle v). although asymmetry can be tolerated in parser evaluation if outputs are compared against gold graphs in a standardized manner, it is difficult to apply an asymmetric metric to measure iaa or to compare parses for detecting paraphrases, or in tri-parsing, where no reference is available. if the asymmetry is amplified by a bias, it becomes harder to judge the scores. finally, considering that sembleu does not match amr graphs on the graph-level but matches extracted bags-ofk-grams, it turns out that it cannot be categorized as a graph matching algorithm as defined in principle vi. principle vi is fulfilled by smatch without any transformation on amr graphs. it searches for an optimal variable alignment and counts matching triples. as a corollary, it fulfills principles i, ii, iii and v. the fact that smatch fulfills all but one principle backs up many prior works that use it as sole criterion for iaa and parse evaluation. our principles also helped us detect a weakness of all present amr metrics: they operate on a discrete level and cannot assess graded meaning differences. as a first step, we propose s 2 match: it preserves beneficial properties of smatch but is benevolent to slight lexical meaning deviations. besides parser evaluation, this property makes the metric also more suitable for other tasks, for example, it can be used as a kernel in an svm that classifies amrs to determine whether two sentences are equivalent in meaning. in such a case, s 2 match is bound to detect meaningsimilarities that cannot be captured by smatch or sembleu, for example, due to paraphrases being projected into the parses.