DRS Parsing as Sequence Labeling (2022)

Abstract: We present the first fully trainable semantic parser for English, German, Italian, and Dutch discourse representation structures (DRSs) that is competitive in accuracy with recent sequence-to-sequence models and at the same time compositional in the sense that the output maps each token to one of a finite set of meaning fragments , and the meaning of the utterance is a function of the meanings of its parts. We argue that this property makes the system more transparent and more useful for human-in-the-loop annotation. We achieve this simply by casting DRS parsing as a sequence labeling task, where tokens are labeled with both fragments (lists of abstracted clauses with relative referent indices indicating unification) and symbols like word senses or names. We give a comprehensive error analysis that highlights areas for future work.

## introduction semantic parsing is the task of mapping naturallanguage sentences to symbolic representations of their meaning. although most current natural language understanding (nlu) applications are handled by end-to-end systems that solve specific tasks (such as machine translation, conversation, or sentiment analysis) without intermediate symbolic meaning representations, semantic parsing continues to attract research interest for good reasons: first, next-generation nlu systems may become more accurate and certainly more easily explainable and debuggable by combining symbolic representations with end-to-end techniques. second, symbolic meaning representations are amenable to symbolic reasoning, which may be instrumental in enabling, e.g., digital assistants to solve more complex tasks. third, better and more transparent computational models of text-meaning mapping can be a useful tool for semantics, i.e., to understand how natural-language semantics works. in recent years, most work on annotating naturallanguage text with comprehensive, broad-coverage meaning representations has been performed in three frameworks: abstract meaning representations @xcite , universal cognitive conceptual annotation @xcite , and discourse representation structures @xcite . accurate parsers exist for all three (e.g., @xcite @xcite . each formalism has its specific strength: amrs go very far in abstracting away from surface variation in how a certain meaning is expressed, ucca has a clear mapping between form and meaning and a modular architecture, and drss ground natural language meaning in first-order logic, by explicitly representing the scopes of negation, quantification, disjunction, etc. in this paper, we focus on parsing to drss. state-of-the-art drs parsers follow the encoderdecoder paradigm pioneered for machine translation by @xcite : the input sequence is encoded by a neural network into a vector, then another network predicts the output sequence (or in this case: output drs) from that vector. rather than improve upon the accuracy of such parsers on standard benchmarks, our aim in this paper is to achieve some of their benefits (ability to learn from examples, high accuracy, low computational complexity, robustness to atypical input, utilization of off-the-shelf language models, conceptual simplicity) while also having a degree of compositionality, traditionally a property of grammar-based systems. specifically, our system learns to assign each token of an utterance one of a finite set of abstract meaning fragments that are deterministically combined to give the meaning of the whole utterance. while our system may not fulfill all criteria of compositionality according to some definitions, it can t1 e1 b2 time.n. @xcite t1 name(x1, "tom") male.n.02(x1) figure 1: drs for the sentence "tom was tricked" in box notation arguably reap some of compositionality's benefits, which make it suitable for use in semi-automatic annotation workflows. we discuss this further in section 5. previous work has introduced trainable compositional semantic parsers for amr @xcite and drs @xcite . in this paper, we improve upon the latter parser using a novel way to encode anchored drss as sequences, and thereby cast drs parsing simply as a sequence labeling task ( §2). we use a standard transformer-based model to learn this task, followed by post-processing to ensure well-formed drss ( §3). we use training data from the parallel meaning bank ( §4). the accuracy of our model approaches the state of the art with the additional benefit of being, to a degree, compositional ( §5). we give an error analysis in §6 and conclude in §7. figure 1 shows the drs for the sentence "tom was tricked" in box notation. it consists of two sub-drss or boxes, b1 and b2. b1 introduces an entity named "tom" x1. b2 introduces a "tricking" event e1 (an event of type trick.v.01 in the wordnet ontology, @xcite ) whose patient role is filled by x1. because "tom" is a definite np, it introduces a presupposition: b2 presupposes b1. the event is in the past, i.e., its time role is filled by a time entity (an entity of type time.n.08 in wordnet) t1 which precedes the time "now". figure 2 shows the same drs in clause notation. here, a drs is a set of clauses. a clause consists of a box label indicating which box the clause is part of, a predicate such as a word sense, our sequence-labeling method assumes training drss to be anchored, that is, each clause must be aligned to one (or more) input token. thanks to the grammar-based annotation method of the pmb, this is approximately the case, as can be seen in the clause representation. we thus encode the drs as a sequence of labels, one for each token, where each label consists of zero or more clauses, as row (1) of figure 3 shows. we call these labels fragments. although labels are complex because they can consist of multiple clauses, our sequence labeling model treats them as atomic. in prediction tasks, it is important that label predictions generalize to unseen data. in contrast to this, the numeric part of referent labels in clauses are not meaningful and depend on the number of referents that were introduced before in the same sentence, so they would generalize poorly. thus, in row (2), we change the referents to be relative, inspired by @xcite : referents that have not occurred before get the index 0 and referents that have occurred get a negative index, indicating how long ago the same referent last occurred (counting back among all occurrences of referents of the same type). to further reduce proliferation of different fragments, we also experiment with factorizing fragments into fragments proper and integration labels. in this factorization, the first backreference of every type in a fragment always has index -1, and a separately predicted integration label specifies how much to subtract from that to get to the actual index. this can be seen in row (3), where the first b label for the word was has index -1 instead of -2, and (0) tom was tricked (1) b1 ref x1 b1 name x1 "tom" b1 presupposition b2 b1 male "n.02" x1 b2 ref t1 b2 tpr t1 "now" b2 time e1 t1 b2 time "n.08" t1 b2 ref e1 b2 patient e1 x1 b2 trick "v.01" e1 the integration label [b-1 e0 n0 p0 s0 t0 x0] indicates that 1 should be subtracted from that to get to the actual relative index. this allows was in our example to have the same fragment as in someone was tricked, where the subject does not introduce a presupposition and the actual index is thus -1 rather than -2 because there is one less box intervening. 2 another important factorization concerns largeclass and open-class symbols, viz. (content-word) word senses, names, numbers, and time expressions. we follow @xcite in replacing these with dummy expressions in the fragments and predicting them separately, as explained below in section 3. we also follow them in heuristically changing the representation of first and second person pronouns, which introduce "speaker" and "hearer" constants instead of discourse referents in the pmb, for more consistent representation of predicates. ## parsing model our parsing model consists of a standard transformer sequence labeling model, followed by postprocessing to assemble the predicted labels into a drs. sequence labeling transformer model our model is schematically depicted in figure 4 . it takes an input sequence of tokens @xmath0. . . w n and produces aligned output sequences y s , y f , y i , which are word senses, fragments, and integration labels. our model simply consists of a pre-trained bert model @xcite and three linear classifiers. each classifier can be seen as a sub-system of the semantic parser that produces one of the three labels (word sense, fragment, and integration label). each input token must further be tokenized into wordpieces before they can be fed into the bert model. to obtain a single representation for a token that consists of n wordpieces and thus produces n embedding vectors, we experiment with two commonly used strategies: taking only the first wordpiece, or averaging the embeddings of all n wordpieces. post-processing after the neural model predicts a fragment and a word sense for each token, we assemble these predictions into a complete clause list by choosing unique new names for discourse referents with index 0 and unifying other discourse referents with them according to their relative indices. we also replace dummy strings in clauses by the predicted word senses and by symbols for names, cardinalities, and date/time expressions, which are predicted from the tokens by a rule-based system similar to that of @xcite . for example, for the proper name tom it predicts the symbol "tom", for the numeral two it predicts "2", and for the time expression five o'clock, it predicts "17:00". special clauses like b1 "speaker" x1 and b1 "hearer" x1 are removed and the corresponding referent (x1 in the example) replaced by the symbols "speaker" and "hearer". finally, we use a set of postprocessing rules similar to that of @xcite to ensure the validity of the resulting drs: if there is a loop in the subordination relation among boxes, an arbitrary box in the loop is chosen, and all its clauses are removed to break the loop (cf. figure 9 and three annotation statuses: gold drss have been fully corrected by human annotators, silver ones have been partially corrected, and bronze ones are the unchecked outputs of rule-based pre-annotation. plms and hyperparameters the backbone of our pytorch @xcite implementation is the transformer and wordpiecetokenizer classes offered by hugging face @xcite . we use pre-trained bert models provided on huggingface.co: bert-base-cased, dbmz/bert-base-german-cased, dbmz/bert-base-italian-cased, and geotrend/bert-base-nl-cased @xcite , keeping their default configuration. the only hyperparameters we choose ourselves are the batch size (24), the learning rate, and the number of epochs. we used the adam optimizer to train all the parameters in our model including the pretrained bert. to ensure stability and avoid overfitting, we used a linear scheduler with no warm-up step, which gradually reduces the learning rate from 0.0015 to 0 for each training iteration. during preliminary experiments on the development set, we found that training loss barely changed after five epochs. bert has 12 layers, each of which has a 768-dimensional output embedding per wordpiece. there is some mixed information in the literature as to which layer's output is most suitable for seman2020 claim that the last layer provides the best results for their drs parser, so we experimented with both. evaluation we evaluate the performance of our parser using counter @xcite , an extension of the smatch evaluation metric @xcite . counter approximates an optimal mapping between the referents in the gold drs and the predicted drs using hill-climbing, then outputs recall, precision, and f-score for the predicted clauses compared to the gold clauses. ## results and discussion integration labels we trained an initial model on the english gold training data, for which we give some statistics in ## error analysis we were interested in which semantic phenomena present particular challenges to our parser and thus performed an error analysis of the output of our best model on the english development data, shown in to gain a better understanding of common error types, we did an exploratory manual analysis, randomly sampling 100 drss produced by our best model on the english development set. thanks to the compositional model structure, we could easily replicate the pmb-style word-clause alignment in the output, which makes these analyses much easier. the examples we refer to can be found in the appendix. in the sample, the most common errors we found were incorrect word senses, for which 36% of the sample drss had at least one instance. the second is semantic roles and discourse relations (30%). despite our intention to separate them into two different sub-tasks, in our sample, these two error types often co-occur (cf. appendix, figure 6 ). in fact, in our sample, we could not find a single case where the predicted word sense of a verb and the predicted semantic roles are not compatible with each other. we hypothesize that correlations between both are learned well by the underlying bert model, which informs both the fragment classifier and the word sense classifier. in a sense, word sense errors could be expected to be much more frequent than semantic role errors, because word senses form a larger class than verbal fragments. it could be that our model tends to produce internally "consistent" meanings (with matching senses and roles) even at the price of predicting incorrect roles, for which it is penalized, since counter does not reward consistency but only correctness. we leave a closer investigation of this hypothesis to future work. compared to verbs, noun fragments have less variation, thus we generally observe fewer errors with them. however, there is a noun-related error that consistently occurs in our sample, viz. failure to recognize demonyms as such and assign them the corresponding analysis, which involves a presupposed country (cf. appendix, figure 7 ). our parser also consistently fails to recognize generic you as opposed to deictic you (cf. appendix, figure 8 ), which points to the importance of discourse context for understanding the (speaker) meaning of even a single word, and perhaps to something that all current drs parsers lack: an explicit distinction between sentence meaning and speaker meaning (cf. @xcite . besides the very large class of word senses, there is also the completely open classes of symbols: names, cardinalities, and times. our parser predicts them from the corresponding tokens using rulebased heuristics, which we have only implemented for english for now. simply copying the token often gives the correct symbol, which is partly why we only saw a 1% difference for them in the previous evaluation and why other languages still have acceptable f-scores (the other reason being that counter arguably underpenalizes incorrect symbols). of course, things can also go wrong (cf. appendix, figure 7 ). finally, we look at fragment predictions with incorrect discourse referent indices, which lead to incorrectly unified discourse referents in the output. the tendency in our sample seems to be that things here go right most of the time, but when they go wrong, they go very wrong, leading to drss that are not just incorrect but invalid and can thus not be scored by counter. one way for a drs to be invalid is to have a loop in its subordination relation, e.g., when two boxes presuppose each other. the way our repair heuristics fix this is to completely delete one of the boxes, and then fix unintroduced referents by introducing new ref clauses, and fix a nonconnected subordination relation by introducing continuation relations between boxes (cf. appendix, figure 9 ). although a bit crude and drastic, these fixing heuristics seem to hurt f-score less than one might expect, for they mainly affect drss that were quite wrong to begin with. ## conclusions we have presented the first fully trainable drs parser that is both competitive with the state of the art and compositional. unlike sequence-tosequence models it provides an explicit mapping between tokens and clauses, and fixed fragments ensure consistent analyses. unlike traditional pipelines, it does not make use of explicit syntactic representations or λ-expressions but uses a simple sequence factorization, and wraps up much of the complexity in a general-purpose bert model. we argue that these characteristics make our model especially suitable for interactive annotation with humans in the loop, but is also good enough for other applications. beyond producing more and better data, our error analysis suggests that the next frontier in drs parsing will involve better modeling of discourse context, and perhaps an explicit separation of sentence meaning and speaker meaning.