The evaluation of whole-sentence semantic structures plays an important role in semantic parsing and large-scale semantic structure annotation. However, there is no widely-used metric to evaluate wholesentence semantic structures. In this paper, we present smatch, a metric that calculates the degree of overlap between two semantic feature structures. We give an efficient algorithm to compute the metric and show the results of an inter-annotator agreement study.
Cai and Knight REF developed a metric named smatch that calculates the degree of overlap between two AMR structures.
Smatch: an Evaluation Metric for Semantic Feature Structures
ACL
Computer Science