Rule-based semantic interpretation for U niversal D ependencies (2023)

Abstract: In this paper, we present a system for generating semantic representations from Universal Dependencies syntactic parses. The foundation of our pipeline is a rule-based interpretation system, designed to be as universal as possible, which produces the correct semantic structure; the content of this structure can then be filled in by additional (sometimes language-specific) post-processing. The rules which generate semantic resources rely as far as possible on the UD parse alone, so that they can apply to any language for which such a parse can be given (a much larger number than the number of languages for which detailed semantically annotated corpora are available). We discuss our general approach, and highlight areas where the UD annotation scheme makes semantic interpretation less straightforward. We compare our results with the Parallel Meaning Bank, and show that when it comes to modelling semantic structure, our approach shows potential, but also discuss some areas for expansion.

## introduction aside from the theoretical interest in discovering how syntactic information contributes to semantic interpretation, there are also a number of practical benefits to augmenting syntactic descriptions with semantic representations. a suitably rich semantic representation automatically makes possible a number of common downstream tasks such as named entity recognition, information retrieval, machine translation, and natural language inference. in this paper, we report on our system for using universal dependencies syntactic annotations (ud: @xcite to produce semantic representations, in this case discourse representation structures (drss: @xcite . figure 1 shows the ud parse and a possible drs representation for a simple sentence. in particular, and unlike much of the state of the art, our pipeline makes heavy use of a rule-based component. this component inspects the ud graph and uses it to produce a number of meaning constructors, the basic building blocks of semantic composition in glue semantics (glue: @xcite . meaning constructors are pairs, the first element of which is a lambda expression in some meaning language, and the second element of which is a formula in linear logic @xcite that expresses a type. the atoms of this linear logic statement are indexed with node labels, thereby anchoring (or 'gluing') the semantics to the syntax. this flexible approach to meaning composition allows each word to make any number of distinct meaning contributions, and frees composition from word order, making it a perfect fit for a dependency grammar like ud (see @xcite . rules in our rule-based system consist of two parts: on the left-hand side, a description which nodes in the ud tree might satisfy (e.g. referring to the node's dependency relation, its lemma, or its features), and on the right-hand side, a meaning constructor to be introduced. this system has been implemented, using a haskell script to inspect the ud tree node by node, comparing each one to the rules in our ruleset, and introducing the appropriate meaning constructor each time a node matches a description (for more details on this process, albeit in a different syntactic setting, see @xcite . 1 once a collection of meaning constructors has been obtained, they are passed to the glue semantics workbench @xcite , which uses them to produce a linear logic proof (or proofs, in the case of scope ambiguities) whose conclusion is the meaning constructor corresponding to the semantic representation of the sentence. we subequently use the python natural language toolkit (nltk: @xcite to perform any post-processing steps, including producing human-and machine-readable drs outputs. our system is part of an ongoing project on universal semantic parsing, and so another prominent feature of our system is its focus on broad coverage. this sets it apart from other works which combine symbolic and machine-learning approaches (e.g. @xcite , since these are limited to specific languages, e.g. english, because specific tools exist, or to other languages for which there exist sufficient data to train a deep learning system. because of the lack of semantically annotated training data for the majority of the world's languages, recent efforts in broad coverage semantic parsing (e.g. @xcite ) have been based on machine translation into english, followed by semantic parsing and projection of the result onto the source language. however, state of the art machine translation is only available for high-resource languages @xcite and is likely to introduce noise even in the best of cases, especially if the languages are typologically distant. instead of this translational approach, we try to leverage ud representations to achieve universality. as far as possible, our rule system produces meanings based exclusively on the ud parse, without invoking language-or lemma-specific rules. section 2 discusses the kinds of rules used in more detail (and section 3 identifies some issues that arise which are of potential relevance to ud as a framework). however, this language-neutral approach means that the output of the rule-based component is necessarily underspecified, since, for example, semantic roles (agent, patient, etc.) do not stand in a one-to-one correspondence with syntactic re-lations (nsubj, obj, etc.). for some languages, this is as far as we can go. but where languages have more resources available, and we can therefore access the language-and lemma-specific information needed, we can make use of various postprocessing steps to further refine our semantic representations. one of these systems, used to convert syntactic labels for dependencies into appropriate semantic role labels, is described in section 4. in section 5, we compare the output of our pipeline with an existing benchmark of drs parsing, the parallel meaning bank (pmb: @xcite . our goals are slightly different from those of the pmb, so although this comparison offers indications about the adequacy of our rule system, it does not offer a perfect gold standard. moving forward, we have further plans for postprocessing, and these are discussed in section 6. we also indicate some limitations of the pmb dataset as a gold standard for drs parsing. ## rules for semantic interpretation by using a rule-based system, we can more easily import insights from theoretical linguistics into automatic semantic interpretation. these insights are generally of a structural nature: e.g. the fact that the logical structures produced by different quantifiers do not straightforwardly match their syntactic structure is the sort of thing that may be difficult for a machine-learning algorithm to infer. 2 at the same time, our universal goals mean that languagespecific information, such as the semantic roles a predicate assigns to its arguments, must be abstracted away from, since we cannot retrieve this information from the ud parse alone. the target output of our rule-based system is therefore not a fully-specified drs. instead, we aim to produce a structurally accurate drs, where the correct discourse referents are present and the hierarchical relations between them are correct; the content of the drs, by which we mean the labels for the relations, or the word senses attributed to the discourse referents, will be filled in only later, by languagespecific post-processing. concretely, except for in the cases where no language-/lemma-specific information is required to determine the correct labels, our rule system outputs syntactic (rather than semantic) labels for the relations between discourse referents, and uses lemmas in place of word senses. ## post-processing a full semantic representation contains many types of information that simply cannot be extracted from the ud tree, even with the aid of linguisticallyinformed rules. typically, this is information that would be associated with lexical entries rather than with structural syntax. the most prominent example of this kind is the mapping from syntactic functions to semantic roles: ud gives us labels like nsubj, obj etc., but how these map to roles like agent, patient, stimulus, experiencer, etc. is verb-specific. we resolve this mapping in a separate post-processing step, where for english we rely on verbnet @xcite , which provides details of the syntactic frames of english verbs and their associated semantic roles. verbnet arguments are specified in terms of syntactic categories with associated selectional restrictions, which we translate into regular expressions over relations resulting from our ud translationsbasically syntactic roles or prepositions. figure 3 shows our translations of some of the frames that verbnet version 3.3 specifies for the verb look. to choose the verbnet frame to use, we pick the frame that has the fewest items not present in the drs; if there is a tie, we reject all frames that do not specify core relations (nsubj, csubj, obj, iobj, xcomp, ccomp) that are present in the drs, and pick the remaining one that has the fewest relations in the drs not present in the frame; and if it still not unique, we keep both options. notice that we minimize elements in the frame not present in the drs before the opposite, because the drs will in many cases contain adjunct relations that are not specified by verbnet frames. as an example, consider the sentence how do people look at and experience art?. in our translated drs, the looking event bears three relations: an nsubj relation (to the discourse referent of people), an at relation (to the discourse referent introduced by art), and a how relation (to some discourse referent (a state) whose identity is asked for). of the frames in figure 3 , we choose the second one, because it specifies two elements that are both present in the drs, whereas the other two frames contain elements that are not in the drs. none of the frames tell us anything about the how relation, which should ideally be spelled out as manner, so this must be resolved in a different way. ## comparison with the pmb in order to assess how well our rule-based system performs, we conducted some experiments comparing our outputs to the german, english, italian, and dutch gold standard datasets (produced and checked by human annotators) provided by the parallel meaning bank v. 4.0.0 @xcite . we compare the pipeline output with the test sets of these languages using the counter tool (van noord, 2022), which enables the comparison of two drss that are expressed in a machine-friendly format called 'clause notation' (see @xcite for details about this notation). we use the automatic parser stanza (version 1.4.0 @xcite to produce the universal dependencies representations which serve as input to our pipeline. in the clause notation, lexical concepts are referred to via their wordnet synset @xcite ; e.g. the concept expressed by the lemma man might be represented as 'man "n.01"'. at present, our pipeline does not deal in this level of lexical granularity, instead simply outputting lemmas as drs conditions. for the purpose of comparison, we therefore assign all lexical concepts a default wordnet sense, suffixing all such conditions with "n.01". our first consideration is coverage. there are a number of cases where the pipeline fails to produce a drs for a given sentence, and therefore comparison with the pmb would be unilluminating. there are three main causes: '(csubj|nsubj)': 'agent', 'over': 'location', 'through': 'location', 'into': 'location' '(csubj|nsubj)': 'agent', 'prep': 'theme' '(csubj|nsubj)': 'agent', 'prep': 'location', 'for': 'theme' figure 3 : select translated verbnet frames for look 1. faulty input: sometimes stanza fails to produce a sensible input for the pipeline. for example, stanza sometimes incorrectly interprets '.' in german ordinal number expressions as the end of a sentence, and therefore produces inappropriate and often ungrammatical parses. for this reason, we omit from our comparisons those sentences where we fail to produce a drs. coverage ranges from 79-93% -see there are two main reasons for the discrepancy between the english and non-english scores. firstly, the pmb uses english synsets for all languages, whereas our pipeline uses lemmas for the equivalent conditions, and these are not translated (e.g. where an italian text uses uomo, we will produce a drs condition 'uomo "n.01"', whereas the pmb gold will have 'man "n.01"'). this means that we will systematically score worse in non-english languages, since almost every single condition which comes from a lexical concept will differ from the pmb gold, and so be scored down in comparison. secondly, we have only implemented the semantic role labeling step described in section 4 for english, so once again the non-english languages contain a number of systematic errors: all relations between discourse referents will be wrong since they will have syntactic rather than semantic labels. the english scores, though, are still not particularly impressive. however, there are a number of things being compared here which we make no effort to cover, and so are bound to do badly on. for instance, we make no effort to find the correct synset for lexical concepts. since our focus is on obtaining the correct semantic structure, a more illuminating comparison would be to compare the structures of our drss, ignoring specific role or concept labels. this was achieved using counter's -dr (default role), -dc (default concept), and -dse (default word sense) flags, which overall ignores the effect of getting role, concept, or word sense/synset labels incorrect. this enables us to compare drss in purely structural terms (along with non-language-specific, discourse-related conditions like presupposi-tion), without worrying about the content of the relations or lexical concepts introduced. f1 scores for this comparison are shown in the second part of our pipeline still performs slightly better on english even in this structural-only setting, which is likely due to the fact that we have so far used english as our primary test language during development of the rules. on the positive side, the fact that performance is quite even across the other three languages, and not much lower than english, shows that our system generalises nicely outside of english. however, it would of course be nice to have gold data from less typologically similar languages to test this further. previous work on drs parsing with neural methods has reported f1 scores in the high 80s on the pmb data (see @xcite ). the results of our rule-based pipeline may seem abysmal in comparison, therefore. however, further testing has shown that the rule-based system degrades less as sentence length increases, and may therefore be more robust. most sentences in the pmb test set are very short: the vast majority are shorter than ten tokens, and the average length is 6.7. to test performance on longer sentences, we annotated wikipedia text from the gum corpus @xcite with the pmb tool. the average sentence length in this dataset is 19.5 tokens. taking into consideration only data for which a non-zero f1 score is obtained (around 80% of the data for the dl models, and around 60% for our pipeline), figure 6 shows the f1 scores for our pipeline with gold ud (ud), our pipeline with automaticallygenerated ud (stanza), a neural parser with no pre-training (no-pt), and a neural parser with the pre-trained bert_base_cased (bert_cased) language model. 4 the neural approaches suffer a major drop in performance compared with the pmb data, while our system suffers a less pronounced degradation. we believe this gives us reason to believe that as our system's rule coverage is improved, performance on both datasets will improve commensurably. ## going beyond the pmb along with improving our rule coverage to bring us more closely in line with the pmb, there are other areas where we intend to go beyond the pmb and capture additional phenomena via further postprocessing steps. presupposition is one such area: at present, the drss included in the pmb are in the format of projective discourse representation theory (pdrt: @xcite , and so in principle have access to a rich set of tools for handling the similarities and differences between the information status of various types of projective content such as presupposition, anaphora, and conventional implicature. setting anaphora to one side for the moment, the projective content the pmb currently contains consists of presuppositions triggered by proper names, definite descriptions, pronouns, and possessives @xcite . our pipeline also currently captures these, since their triggering contexts are legible in the ud parse (assuming suitable part of speech tags and lexical features). we are currently conducting experimental work to determine other presupposition triggers which can be incorporated into our pipeline, some perhaps on a language-specific basis. the lack of more presuppositional content in the pmb means it does not at present live up to the potential afforded by its pdrt underpinnings. anaphora resolution is an essential step in semantic interpretation, which not only changes the labelling of a drs, but also affects its structure. we do not currently implement any anaphora resolution, whereas the pmb does, so it may be that this is another area where our scores have been dragged down. however, since the pmb dataset consists of fairly short sentences, there will be fewer opportunities for this to make a significant difference. what is more, the pmb's anaphora resolution is apparently fairly unsophisticated, and linguistically naïve: for example, it violates well-established constraints on binding, as in this principle b violation from the english test data: tom i never spoke of him i . a more robust anaphora resolution system would therefore improve the performance of our pipeline beyond the level of the pmb. ## conclusion we have presented a pipeline for converting universal dependencies parses into semantic representations in the form of drss. our rule-based system is intended to provide as linguistically broad a coverage as possible, producing semantic structures which faithfully capture the relations between discourse referents encoded in syntactic structure. sometimes the ud parse itself is the cause of friction, and we have suggested some ways in which the ud framework might be improved so as to reduce the difficulty of semantic interpretation. since our rule system produces underspecified drss, we also discussed one example post-processing step used to enhance and fully specify our representations. comparison with the pmb shows that in terms of raw coverage we still have a way to go, but that our goal of capturing universal structural information is on the right track, insofar as our rules seem to generalise across the four languages represented in the pmb to similar extents. since we now have a successfully implemented system and a working framework for evaluation, we have laid the groundwork for further progress to be made on a theoretical level with regard to improving and expanding the coverage of our ruleset.