D scorer: A Fast Evaluation Metric for Discourse Representation Structure Parsing (2020)

Abstract: Discourse representation structures (DRSs) are scoped semantic representations for texts of arbitrary length. Evaluating the accuracy of predicted DRSs plays a key role in developing semantic parsers and improving their performance. DRSs are typically visualized as boxes which are not straightforward to process automatically. Counter transforms DRSs to clauses and measures clause overlap by searching for variable mappings between two DRSs. However, this metric is computationally costly (with respect to memory and CPU time) and does not scale with longer texts. We introduce Dscorer, an efficient new metric which converts box-style DRSs to graphs and then measures the overlap of n-grams. Experiments show that Dscorer computes accuracy scores that are correlated with Counter at a fraction of the time.

## introduction discourse representation theory (drt) is a popular theory of meaning representation @xcite @xcite designed to account for a variety of linguistic phenomena within and across sentences. the basic meaning-carrying units in drt are discourse representation structures (drss). they consist of discourse referents (e.g., x 1 , x 2 ) representing entities in the discourse and conditions (e.g., male.n.02(x 1 ), agent(e 1 , x 1 )) representing information about discourse referents. every variable and condition are bounded by a box label (e.g., b 1 ) which implies that the variable or condition are interpreted in that box. drss are constructed recursively. an example of a drs in boxstyle notation is shown in figure 1(a) . drs parsing differs from related parsing tasks (e.g., @xcite in that it can create rep-resentations that go beyond individual sentences. despite the large amount of recently developed drs parsing models @xcite @xcite @xcite , the automatic evaluation of drss is not straightforward due to the non-standard drs format shown in figure 1 @xcite . it is neither a tree (although a drs-to-tree conversion exists; see @xcite for details) nor a graph. evaluation so far relied on counter @xcite which converts drss to clauses shown in figure 1(b) . given two drss with n and m (@xmath0, counter has to consider n! (n-m)! possible variable mappings in order to find an optimal one for evaluation. the problem of finding this alignment is np-complete, similar to other metrics such as smatch @xcite for abstract meaning representation. counter uses a greedy hill-climbing algorithm to obtain one-to-one variable mappings, and then computes precision, recall, and f1 scores according to the overlap of clauses between two drss. to get around the problem of search errors, the hill-climbing search implementation applies several random restarts. this incurs unacceptable runtime, especially when evaluating document-level drss with a large number of variables. another problem with the current evaluation is that counter only considers local clauses without taking larger window sizes into account. for example, it considers "b 4 sing e 2 " and "b 3 not b 4 " as separate semantic units. however, it would also make sense to assess " b 3 not b 4 sing e 2 " as a whole without breaking it down into smaller parts. by considering higher-order chains, it is possible to observe more global differences in drss which are important when assessing entire documents. in order to address the above issues, we propose dscorer, a highly efficient metric for the evalu-ation of drs parsing on texts of arbitrary length. dscorer converts drss (predicted and gold) to graphs from which it extracts n-grams, and then computes precision, recall and f1 scores between them. the algorithm operates over n-grams in a fashion similar to bleu @xcite and rouge @xcite , which are metrics widely used for evaluating the output of machine translation and summarization systems. while bleu only calculates precision with a brevity penalty (it is not straightforward to define recall given the wide range of possible translations for a given input), rouge is a recall-oriented metric since the summary length is typically constrained by a prespecified budget. foot_0 however, in drs parsing, there is a single correct semantic representation (goldstandard reference) and no limit on the maximum size of drss. our proposed metric, dscorer, converts box-style drss to a graph format used for evaluation and computes f1 with high efficiency (7,000 times faster compared to counter). we release our code, implementing the metric, at https: //github.com/leoncrashcode/drsscorer. ## dscorer the proposed metric converts two box-style drss into graphs, extracts n-grams from these graphs, and then computes precision, recall, and f1 score based on the n-gram overlap. ## experiments in our experiments, we investigate the correlation between dscorer and counter, and the efficiency of the two metrics. we present results on two datasets, namely the groningen meaning bank (gmb; @xcite ) and the parallel meaning bank (pmb; @xcite . we compare two published systems on the gmb: drts-sent which is a sentence-level parser @xcite and drts-doc which is a documentlevel parser @xcite . on the pmb, we compare seven systems: boxer, a ccg-based parser @xcite , amr2drs, a rule-based parser that converts amrs to drss, sim-spar giving the drs in the training set most similar to the current drs, spar giving a fixed drs for each sentence, seq2seq-char, a character-based sequence-tosequence clause parser @xcite , seq2seq-word, a word-based sequence-to-sequence clause parser, and a transformer-based clause parser @xcite . ## related work the metric sembleu @xcite is most closely related to ours. it evaluates amr graphs by calculating precision based on n-gram overlap. sembleu yields scores more consistent with human evaluation than smatch @xcite , an amr metric which is the basis of counter. sembleu cannot be directly used on drs graphs due to the large amount of indexed variables and the fact that the graphs are not explicitly given; moreover, our metric outputs f1 scores instead of precision only. @xcite propose a set of principles for amr-related metrics, showing the advantages and drawbacks of alignment-and bleu-based amr metrics. however, efficiency of the metric is crucial for the development of document-level models of semantic parsing. @xcite propose to represent drss via discourse representation graphs (drgs) which are acyclic and directed. however, drgs are similar to flattened trees, and not able to capture clause-level information (e.g., b 1 agent e 1 x 1 ) required for evaluation @xcite . ## conclusions in this work we proposed dscorer, as a drs evaluation metric alternative to counter. our metric is significantly more efficient than counter and considers high-order drss. dscorer allows to speed up model selection and development removing the bottleneck of evaluation time.