Evaluate AMR Graph Similarity via Self-supervised Learning (2023)

Abstract: In work on AMR (Abstract Meaning Representation), similarity metrics are crucial as they are used to evaluate AMR systems such as AMR parsers. Current AMR metrics are all based on nodes or triples matching without considering the entire structures of AMR graphs. To address this problem, and inspired by learned similarity evaluation on plain text, we propose AMRSim, an automatic AMR graph similarity evaluation metric. To overcome the high cost of collecting human-annotated data, AMRSim automatically generates silver AMR graphs and utilizes self-supervised learning methods. We evaluated AMRSim on various datasets and found that AMRSim significantly improves the correlations with human semantic scores and remains robust under diverse challenges. We also discuss how AMRSim can be extended to multilingual cases.

## introduction an abstract meaning representation (amr; @xcite is a rooted, directed graph where nodes represent concepts and edges correspond to relations of concepts. the goal of an amr metric is to evaluate the similarities of pairs of amr graphs so that it can be used to evaluate the outputs of amr generators such as parsers. therefore, a good amr metric is crucial for the design and evaluation of amr parsers. however, the research on amr metrics has so far lagged far behind the work on amr parsing. current amr metrics either transfer amr graphs to triples and consider the one-to-one matching of variables @xcite or linearize amr graphs as sequences and calculate n-gram matching @xcite . these metrics fail to consider the entire amr structure and lack flexibility, resulting in poor correlation with human annotations. inspired by plain-text automatic similarity assessment methods that encode sentences into latent semantic representations to measure a similarity of the two representations @xcite , we propose to learn the automatic assessment of amr graph similarity through a similar pipeline. our proposed metric, called amrsim, adopts the pre-trained language model bert as the backbone and incorporates gnn adapters to capture the structural information of amr graphs. to overcome the high cost of collecting training data, we utilize self-supervised learning methods. the training objective is to maximize the dot product between positive embeddings and to minimize the dot product between different encodings. in contrast to one-to-one matching metrics, our amr-sim is alignment-free by computing the cosine of contextualized token embeddings. the prediction process can be considerably accelerated by leveraging gpus. we experiment with amrsim on the transformed stsb @xcite and sick @xcite datasets, which contain pairs of amr graphs and corresponding similarity scores. our experiments demonstrate that amr-sim achieves prominent improvements in correlation with human annotations. in further analysis, amrsim retains the highest performance under various challenges compared to previous metrics. we also explore the potential of extending our metric to multilingual cases, taking into account the generic nature of the transformer structure and the fact that the pipeline of amrsim is not constrained to any particular language. the remaining paper is structured as follows. section 2 gives an overview of existing amr metrics; section 3 introduces our proposed metric, amrsim; section 4 includes experimental settings and main results; we analyze the robustness and the efficiency of our metrics in section 5 and conclude in section 6. ## existing metrics amr similarity metrics play a vital role in evaluating the performance of amr parsers. however, computing the degree of similarity between amr graphs is not trivial. in this section, we summarise the existing amr metrics. smatch smatch @xcite evaluates the overlap of structures as the similarity score. each amr graph is viewed as a conjunction of triples. smatch tries to find a one-to-one matching of variables that maximizes the numbers of exact matches of triples through the hill climbing method in a greedy style. the alignment process, which has been proved as np-hard, limits the efficiency of smatch, especially as the sizes of amr graphs increase. another weakness is that the structural matching is insufficient for meaning similarity assessment and fragile in concept synonym replacement and structure deformation @xcite . s 2 match to yield a better variable alignment, @xcite propose s 2 match by allowing soft semantic match instead of matching only identical triples in smatch. accounting for cosine similarity of concepts helps to assess the similarity of graphs, however, computational limitations and structure deformation confusion in smatch have not been addressed. sema smatch adds a top relation to the structure, but sema @xcite argues that this addition can potentially distort the evaluation. therefore, they ignored triples identifying the graph top. furthermore, instead of computing the maximum score like smatch, sema works as a breadth-first search and produces a deterministic result, making it faster than one-to-one matching of variable employed by smatch. sembleu bleu @xcite , which assesses text quality by comparing n-grams, is frequently adopted in machine translation evaluation. to extend bleu for matching amr graphs, sem-bleu @xcite linearizes amr graphs through breadth-first traversal and extracts n-gram for comparison. the metric is alignmentfree and thus computationally efficient. experimental results show that sembleu achieved slightly higher consistency with human judgment than smatch. wwlk besides treating amr graphs as triples and linearized grams, weisfeiler-leman amr similarity metrics @xcite consider amr graphs as high-dimensional objects. they first propagate node embeddings iteratively by incorporating contextualization and then employ wasserstein weisfeiler leman kernel (wwlk) to calculate the minimum cost of transforming one graph to another. wwlk only considers node embeddings (glove embeddings) in amr graphs, while edge labels have no corresponding embeddings. therefore, wwlk θ is extended to learn amr edge labels, which requires additional training data. considering embeddings of amr edges improves the performance of metrics. however, supervised learning methods require additional efforts to collect human-labeled data. in contrast, our proposed learned amr similarity metric adopts selfsupervised learning and achieves higher correlation performance. ## the proposed approach in this section, we introduce our proposed approach for learning amr graphs similarity evaluation. ## analysis amrsim shows a high correlation with human annotations. in this section, we further analyze the robustness and efficiency of amrsim.