Bridging Semantic Frameworks: mapping DRS onto AMR (2023)

Abstract: A number of graph-based semantic representation frameworks have emerged in recent years, but there are few parallel annotated corpora across them. We want to explore the viability of transforming graphs from one framework into another to construct parallel datasets. In this work, we consider graph rewriting from Discourse Representation Structures (Parallel Meaning Bank (PMB) variant) to Abstract Meaning Representation (AMR). We first build a gold AMR corpus of 102 sentences from the PMB. We then construct a rule base, aided by a further 95 sentences. No benchmark for this task exists, so we compare our system’s output to that of state-of-the-art AMR parsers, and explore the more challenging cases. Finally, we discuss where the two frameworks diverge in encoding semantic phenomena.

## introduction many semantic representation frameworks have emerged over the years @xcite @xcite , at varying levels of abstraction in terms of encoding semantic phenomena. we want to be able to compare frameworks empirically across phenomena, with the goal to understand, unify and extend them. unfortunately, this is difficult to do in a data-driven manner as there are few freely available parallel datasets. as manual annotation is laborious, it is important to develop automatic tools to create and expand datasets. one way to approach this is by transforming annotations across frameworks. in this work, we take a look at abstract meaning representation (amr) @xcite , and discourse representation structures (drs) @xcite , as expressed in the parallel meaning bank (pmb) @xcite , to see how much of the former can be constructed from the latter. we show a significant portion of amr can be constructed from drs and provide a discussion on our insights as to where the process is not possible. to achieve this we build a graph rewriting system from drs to amr. as there is no parallel data between the two, we also annotate a small part of the pmb into amr. our motivation for this work is twofold. our first goal is to get more parallel annotated data between semantic formalisms in general, and between amr and drs for this particular study, in order to foster empirical cross-formalism comparison. a natural question to ask here is, since (as we will see in section 5) automatic parsers based on machine learning techniques seem to perform better than rule-based transformation systems on this task, why do we bother with such an experiment. we have a few reasons: (i) rule-based transformation systems may still perform quite well, especially for more closely-related formalisms (as we show in this study) and we do not know how well exactly until we test such a system; (ii) it is possible that the two approaches make different kinds of mistakes, which opens the possibility for hybrid solutions that combine their strengths; (iii) with a rule-based system, tracking the decision-making process is possible, rendering the method explainable. our second goal is to better understand the differences between formalisms with a view to extend and unify them. this is difficult to do in a non-data-driven manner as the formal definitions of formalisms are rarely complete. more importantly, within the community, it is not clear what a complete semantic representation should consist of. thus, while not as direct as the first, an outcome we hope to get from this work is a deeper insight into what is needed in a semantic representation and what are the missing links between formalisms, as a step towards defining a unifying framework. the rest of the paper is structured as follows: in section 2, we present the two frameworks; in section 3 -our graph rewriting system (grs); section 4 is about our annotation procedure for a small gold amr dataset; in section 5, we present our experiments, discuss the results, and compare them to those of sota amr parsers; in section 6, we provide a discussion and future work directions. our code and data are publicly available foot_0 . ## background in this section we present the parallel meaning bank as an instance of a large corpus of discourse representation structures, and abstract meaning representation. ## system we use grew 7 @xcite to build a graph rewriting system (grs) for rewriting sbn graphs into amr ones. grew is a tool that allows the user to define rules to match patterns in a graph and apply a set of commands that transform the matched part of the graph. grew also allows for the use of lexicons, which lets us map sets of values and assign a value to a variable based on the value of another variable. ## gold data to evaluate our system, we produced gold amr annotations for 102 sentences of the english part of the pmb. in order to make sure that there were no specific phenomena concentrated in certain partition of the pmb data, instead of picking a random partition and risking having a non-representative sample, we applied an algorithm to "randomise" that 12 . we created 100 new partitions, by finding the sum of the part and document number of each sentence and applying modulo 100 to get a new partition number. this approach groups the data randomly, but is reproducible and as the pmb expands, the partitions should grow in a fairly uniform manner. version 4.0.0 of the pmb contais 10,715 gold english sentences, so 107 sentences on average per partition. we picked partition 25 (i.e. all the documents for which (p + d)%100 is 25) to annotate manually. it contains 102 sentences. our four annotators -a, b, c and d -annotated half of the sentences (51) each. every sentence was annotated by two annotators. to ensure that each pair of annotators had the same number of overlapping sentences, we split the 102 sentences into six groups of 17 and distributed the groups among the six different pairings. the annotators consulted the following resources during the annotation process: we used smatch @xcite to compute the inter-annotator agreement (iaa). smatch uses a hill-climbing algorithm to find the maximum number of triples between two graphs. there are three types of triples: instance, relation, and attribute. instance triples match nodes in the graph, counting exact matches between the node concepts. relation triples match edges in the graph. attribute triples match properties of the nodes. each type has equal weight in the overall score count. the results of our iaa are reported in to produce the final version of the gold data, the four annotators gathered in groups (two, three, or four) over the course of a few sessions. for each sentence, the two existing annotations were compared and after a discussion, one was chosen or a modification that combines elements of both annotations was selected. in a small number of cases, the annotators agreed on an entirely different annotation from the two proposed ones. ## evaluation as with our iaa, we use smatch to evaluate our system's output against the gold annotations. as mentioned in section 4, smatch takes into account not only the graph structure, but also the exact match of concepts between graphs. thus, we expected that the predicate lexicon and the second post-processing step (removing trailing part of wordnet concepts) would have a substantial impact on the final score. to evaluate this, we run the experiment with no lexicon (1), with the incomplete lexicon (2), and with the complete lexicon (3). additionally, we also run a version with the complete lexicon, but without the second post-processing step (4). finally, while adding a lexicon of senses increases the results for both the test and dev sets significantly, we see that the difference in results between the incomplete lexicon and the complete lexicon setting is very small. our hypothesis is that this is due to the fallback rules for core roles that have not been rewritten. to verify this, we run the three different lexicon versions (no lexicon (5), incomplete lexicon (6), and complete lexicon (7)) also without the fallback rules. we run each of these experiments on both the dev and test sets. the results from our experiments are reported in due to non-deterministic rules, for some sen-tences we get more than one output graph. as we want to see what is the biggest part of the amr structure that can be built from drs, the scores we report take the graph with highest overlap (according to smatch f1-score) with the gold graph 19 . comparing the scores on the dev and test sets, we notice the big disparity in the scores between the two. this is due to our building the rules based on the dev set and thus missing out on structures that do not appear in it, but appear in the test set. this suggests that our rule set is incomplete and a larger dev set may be necessary to ensure broader structure coverage. one such example is that the structure used in the pmb for expressions such as instead of and rather than needs special treatment which requires either the duplication of a specific node or the introduction of the predicate prefer-01. however, since our rule base was built from the dev set and that does not include such an example, we do not have a rule to address it. we do, however, have such an example in the test set and it cannot be addressed properly. ## discussion our goal with this work was to see what portion of amr can be constructed from drs and where that is not possible, to understand why. while constructing our rule base, we observed that the way the two frameworks encode predicate-argument structure is very similar, differing mostly in semantic role labels, where drs relies on verbnet roles and amr on propbank predicates. with an exhaustive lexicon that contains a mapping between all senses and their arguments in the two lexical resources, it will be possible to rewrite these correctly. the most notable difference between the two frameworks is the lack of scope in amr, whereas that is present in drs. some phenomena linked to scope are encoded differently in the two. e.g., universal quantification is typically encoded in the pmb in the same way as generics: the sentences "all the seats are booked." (50/2764) and "a cat has two ears." (60/0913) have a similar structure, despite the different phenomena. in amr the two are encoded differently as the quantifier "all" is present on the surface in one and not in the other. similarly, the quantifiers "the" and "this" are expressed in the same way in drs: by neither being present in the representation, while in amr "this" is expressed as a separate node and "the" is not. drs, as the name suggests, is centered around discourse (as opposed to dialogue) and is not meant to encode questions very well. we observe that in the pmb, wh-questions can be derived from the representation. however, this is not the case for yes-no questions, which, in the pmb are encoded exactly as their declarative counterparts. this is not the case in amr, thus preventing us from dis-figure 3 : amr annotation of the sentence "all who were arrested have been released." (99/1243), the way it would look like if we were to follow the "logical" reading as in drs. figure 4 : amr annotation of the sentence "all who were arrested have been released." (99/1243), the way a human annotator using the amr guidelines and datasets as examples would likely annotate it. that being said, for a number of our rules in the boxes category, we do make use of grew's ability to check for specific regular expressions in the original sentence. this works for short sentences where there is a small or no risk of having a specific structure appear more than once. however, it is not a valid solution for longer texts. a future version of this system may benefit from using the sbn notations comments (part after % in figure 1d ). finally, we want to discuss a broader issue in relation to universal quantification in drs. in the pmb, sentences such as "all who were arrested have been released." (99/1243) have a structure which corresponds to the reading "if a person has been arrested, they have been released". this is the way to express the semantics of universal quantification in logic. it is achieved in the pmb by using a combination of a condition-consequence box embedding. this can be rewritten into amr by making use of the non-core role condition, obtaining the graph in figure 3 . this is a correct reading of the sentence. however, if an annota-tor was to follow the amr guidelines, we would get the graph in figure 4 . we believe this is also a correct representation of the sentence. while logically the two may be equivalent, the graph representations are structurally different. this raises the question of whether we can have more than one correct amr per sentence. if so, then this opens the door for future considerations on how to take that into account in our evaluation metrics. there are a number of other improvements to our system that are worth exploring in the future. expanding the rule base can happen in two main ways (1) by expanding the dev set so that more varying structures are present and (2) thoroughly going though the different expressions in the amr guidelines and amr dictionary and designing rules for each of them. ideally, a combination of the two should be considered. furthermore, as we have seen with our experiments in section 5, having a lexicon that maps wordnet senses to propbank predicates improves the score significantly. our lexicon is still incomplete and can be further improved by adding adjectives, for instance. it would also be interesting to explore how our system performs on other languages (see appendix a). our effort in trying to transform frameworks is not unique for the semantic representations community. in an exploration to better understand what linguistic semantic phenomena formalisms encode, @xcite propose a rule-based conversion system from syntax and lexical semantics into universal conceptual cognitive annotation. closer to our work in terms of formalisms used, bos (2020) proposes amr+ (an amr extension to deal with scope) and a formal procedure to convert amr+ into drs. as a future work, we are interested in seeing how much amrs obtained by applying the reverse procedure (from drs to amr+), then dropping the scope information, would differ from what we obtained with our system. ## conclusion the goal of this work was to build a graph rewriting system from drs (as in the pmb) to amr to discover what portion of the latter can be constructed from the former. to do so, we first constructed a small amr dataset from pmb sentences and built a lexicon mapping wordnet senses to propbank predicates and arguments. we showed a significant part of the amr structure is contained in drs. finally, we discussed their divergences.