Inspecting Soundness of AMR Similarity Metrics in terms of Equivalence and Inequivalence (2024)

Abstract: In this study, we investigate soundness of current Abstract Meaning Representation (AMR) similarity metrics in terms of equivalence and inequivalence. Specifically, AMR guidelines provide several equivalence and inequivalence conditions to reflect the meaning aspect of the semantics. Thus, it is important to examine an AMR metric’s soundness, i.e., whether the metric correctly reflects the guidelines. However, the existing metrics have less investigated their soundness. In this work, we propose a new experimental method using simulated data and a series of statistical tests to verify the metric’s soundness. Our experimental result revealed that all existing metrics such as Smatch, SemBLEU, S2match, Smatch++, WWLK-theta, WWLK-k3e2n, and SEMA did not fully meet the AMR guidelines in terms of equivalence and inequivalence aspects. Also, to alleviate this soundness problem, we suggest a revised metric called Smatch#, which adopts simple graph standardization technique that can improve the soundness of an existing metric.

## introduction in this paper, we propose a new experimental method to evaluate soundness of abstract meaning representation (amr) similarity metrics and try to address the soundness of amr similarity metrics by proposing a revised metric, smatch ♯ . amr is a widely used formalism that expresses the semantic aspect of natural language sentences. the formalism is based on neo-davidsonian semantics @xcite @xcite . therefore, when comparing two amr graphs, a metric needs to yield results that adhere to such theoretical background, which is implemented in the amr guidelines @xcite . we refer to this criterion as the soundness of an amr metric. here, we define soundness as a metric's quality * these authors contributed equally to this work. to yield well-founded results that adhere to the theoretical background of amr during the metric's computation process. for example, soundness of a metric can be operationally checked by whether the metric correctly follows amr guidelines, as amr guidelines define many special equivalence relationships between two amrs with different forms along with its theoretical background. thus, an amr metric should treat such amrs as equivalent to meet the soundness criterion. however, the existing metrics' design has been less focused on evaluating their soundness. several metrics have been proposed to measure the similarity between two amrs, including smatch @xcite , sembleu @xcite , s 2 match @xcite , wwlk θvariants @xcite , sema @xcite , and smatch++ @xcite . although these existing metrics have helped evaluating the quality of various amr parsers, they do not sufficiently consider soundness. the only exception is smatch++, which attempts to address soundness partially by managing some equivalent cases, like reification. nonetheless, even smatch++ has not reported whether their metric adheres to other equivalent cases specified in the amr guideline. therefore, we designed an experiment that investigates the soundness of amr metrics, using systematically simulated data. we implement both 6 equivalent cases and 7 inequivalent cases according to amr guideline, to make a systematic data for evaluating the soundness of metrics. we also propose a simple statistical method to verify soundness and a graph standardization method for handling equivalence and inequivalence cases. as a result, we propose smatch ♯ , an enhanced version of smatch++, as an alternative to prior amr metrics that better addresses soundness. our paper is structured as follows: section 2 provides theoretical background on amr and assesses the designs of existing metrics from the perspective of equivalence and inequivalence. next, sections 3 details our experimental design. specifically, section 3 outlines the simulated dataset generation, the proposed statistical test for soundness verification, the smatch ♯ metric, and implementation details. finally, section 4 presents the results, and discuss their implications. we analyzes the results from applying our experiment to various amr metrics and examines their soundness issues. ## inspecting amr similarity metrics here, we discuss seven existing amr similarity metrics in terms of the way that they handle equivalent and/or inequivalent cases. as widely used similarity metrics adopt a method of giving partial credits to non-exact matching cases, existing metrics differ in how they establish the range of partial credit regarding equivalence and inequivalence of amr components. thus, we categorize the existing metrics into two types: (1) allowing credits only to exact equivalent components, and (2) allowing credits also to some inequivalent cases. first, there are metrics that only give credit for exactly matching/overlapping components when measuring the similarity between two amrs. smatch @xcite , sembleu @xcite , sema @xcite , and smatch++ @xcite belong to this category. these metrics approximately compute the maximum overlap between two amrs, by constructing a mapping between substructures of two amrs. for example, smatch computes overlap as the maximum f 1 score of common triples between two amrs. similarly, in sembleu, the metric computes overlap as the bleu score using n-grams of triples commonly appearing in the two given amrs. however, these overlapbased metrics can mistakenly identify equivalent amrs as inequivalent, since they primarily focus on matching exactly the same components without fully considering the amr guidelines. as the guidelines define some cases where amrs are syntactically inequivalent but semantically equivalent, the soundness of the evaluation may decrease in some cases. second, for the metrics allowing credits also to some inequivalent cases, they try to measure similarity by relaxing the constraint of exact match. metrics such as s 2 match, wwlk θ , and wwlk e2n @xcite @xcite belong to this category. these metrics attempt to give partial credit for inequivalent amrs by incorporating the concept of pragmatic sense, acquired by a language model. with a language model, these metrics are able to construct intuitive sense of similarity between some predicates or between some instances. however, the use of language models makes it difficult to verify that these metrics fairly assess the meaning of amrs independently of any context, which contradicts one of the key assumptions behind amr -that meaning should be context-independent. more specifically, the amr guideline tries to ensure context-independency of semantics by using predefined ontology of predicate senses, semantic roles, and frame arguments from ontonotes @xcite and propbank @xcite . using a language model may compromise such context-independency when comparing two amrs, since a language model tries to treat different pre-defined senses, roles, and arguments as similar ones. moreover, such intuitive sense of similarity may weaken the transparency of the evaluation process. we suspect that all the metrics in the above two categories may insufficiently handle equivalent and inequivalent cases according to the amr guidelines. for example, the case illustrated in appendix a shows that some prior metrics do not correctly evaluate inequivalent amrs which have different meanings. note that goodman (2019) have already shown that not handling these conditions results in an unfair evaluation in smatch. we suspect that other metrics have also insufficiently considered the issues raised by @xcite , because other metrics were not designed to properly handle equivalent and inequivalent cases according to the amr guidelines. moreover, existing metrics have not systematically verified whether they conform to the equivalence/inequivalence conditions based on the amr guidelines. systematic verification of these conditions would therefore be helpful to identify strengths and weaknesses of the existing amr metrics. ## results and discussion second, for the seven inequivalent cases, some metrics showed incorrect evaluation results by assigning ζ max to certain graph pairs, as seen in the p (ζ = 1) row for the inequivalent cases. specifically, sema produced a score of 1.103, exceeding the theoretical ζ max . moreover, among all the metrics, sema was the only one that achieved statistical significance in only 3 cases for the inequivalent cases. we suspect these results from sema appear to be numerical errors caused by its approximation algorithm. furthermore, s 2 match assigned ζ max to some edge deletion pairs (thus, @xmath0.05), and sembleu did the same for some edge insertion pairs (@xmath1.05), while smatch, wwlk e2n , and smatch++ passed all the tests, correctly identifying those cases as inequivalent. third, for the overlap between equivalent and inequivalent cases, all existing metrics showed overlap. for example, as smatch made overlap between equivalent and inequivalent cases on the interval [0.371, 1], the score positioned in this range cannot be determined either equivalent or inequivalent. similar overlap happens for sema (range of [0.083, 1.103]), wwlk θ (range of [0.656, 1]), smatch++ (range of [0.5, 0.998]), and so on. thus, we need to be careful in interpreting a score fall within the overlap range because there may exists incorrect evaluation in terms of amr's theoretical background. even the chance of falling in the overlap range is low, the existence of these overlapping sections is sufficient to pose a question about the soundness of existing metrics. on the other hand, smatch ♯ has proven to be effective in dealing with all the 13 cases. smatch ♯ correctly assigned ζ max in 100% of equivalence cases, achieving the highest possible score, which no other metric accomplished. furthermore, smatch ♯ did not assign ζ max for any inequivalence cases, as confirmed statistically. specifically, for the six equivalent cases, smatch ♯ successfully provide ζ max . for the seven inequivalent cases, smatch ♯ showed slight decrease in score compared to smatch++, the backbone of smatch ♯ . for example, smatch++ had an average score of 0.927 for edge deletion case, while smatch ♯ scored an average of 0.907. moreover, smatch ♯ * alternative hypothesis ha : p (ζ = 1) > 99.9% lift up 1.000 *** .964 .964 .999 1.000 1.000 .990 .949 reorder 1.000 *** .999 .998 1.000 *** 1.000 *** 1.000 *** 1.000 1.000 * relabel 1.000 *** .999 .999 1.000 ** 1.000 *** 1.000 *** 1.000 1.000 *** reify 1.000 *** .748 .748 .613 .866 .864 .660 .990 dereify 1.000 *** .988 .988 .975 .994 .994 .990 .990 duplicate 1.000 *** 1.212 .986 .470 .910 .937 .780 1.000 *** overall min(ζ) 1.000 .371 .371 .025 .610 .628 .083 .500 overall p (ζ = 1)% 100 48.11 55.77 64.74 64.75 64.75 61.47 78.25 7 inequivalent cases * alternative hypothesis ha : p (ζ = 1) < 0.1% insert node .955 *** .975 *** .973 *** .935 *** .952 *** .944 *** .970 *** .966 *** insert edge .964 *** .983 *** .980 ** .931 .996 *** .970 *** .980 *** .977 *** change node .944 *** .966 *** .961 *** .871 *** .940 *** .951 *** .890 .946 *** change edge .949 *** .966 *** .965 .935 *** .982 .968 *** .950 .952 *** delete node .906 *** .945 * .946 *** .908 *** .936 *** .933 *** .940 .918 *** delete edge .907 *** .948 *** .949 .930 *** .959 *** .946 *** .930 *** .927 *** swap .873 *** .918 *** .918 *** .853 .949 *** .954 *** .880 .884 *** overall max(ζ) .998 1.000 1.000 1.000 1.000 0.999 1.103 .998 overall p (ζ = 1)% .00 .01 .05 .65 5.09 .00 .63 .00 + @xmath2.1, * @xmath3.05, ** @xmath4.01, *** @xmath5.001 ## conclusion in this study, we proposed a novel experiment for verifying soundness of an amr metric using simulated dataset and statistical tests. through the experiment, our work demonstrated that the soundness problem exists in the previous metrics. also, we suggest an amr metric smatch ♯ , which is an improved version of smatch++ in terms of soundness, using a graph standardization method that follows amr guidelines. by testing smatch ♯ with the same experiment, we demonstrated that we can alleviate the issue by slightly enhancing the design of metrics. for future work, designing a new amr similarity metric by considering our experimental results would be an interesting topic to pursue.