Universal Decompositional Semantic Parsing (2020)

Abstract: We introduce a transductive model for parsing into Universal Decompositional Semantics (UDS) representations, which jointly learns to map natural language utterances into UDS graph structures and annotate the graph with decompositional semantic attribute scores. We also introduce a strong pipeline model for parsing into the UDS graph structure, and show that our transductive parser performs comparably while additionally performing attribute prediction. By analyzing the attribute prediction errors, we find the model captures natural relationships between attribute groups.

## introduction a structured account of compositional meaning has been longstanding goal for both natural language understanding and computational semantics. to this end, a number of efforts have focused on encoding semantic relationships and attributes in a semantic graph-e.g. abstract meaning representation (amr; @xcite , universal conceptual cognitive annotation (ucca; @xcite , and semantic dependency parsing (sdp; @xcite @xcite . in these formalisms, semantic information is typically encoded discretely, using nominal category labels for nodes and edges. this categorical encoding can make such formalisms brittle when presented with non-prototypical instances, and leads to challenges in coping with changing label ontologies and new datasets @xcite . furthermore, they are difficult to annotate, often requiring trained linguists and large annotation manuals. the decompositional semantics framework presents an alternative to categorical formalisms that encodes semantic information in a featurebased scheme-using continuous scales rather than categorical labels. starting with a feature-based semantic role representation rooted in @xcite proto-role theory @xcite , this framework has expanded to cover a wide variety of phenomena: event factuality @xcite , genericity @xcite , entity types @xcite , and temporal relations @xcite . while this rich array of annotation types has been separately modeled, no system yet exists for its joint prediction, which has only recently been made feasible by the introduction of universal decompositional semantics v1.0 (uds1.0). presented by @xcite , uds1.0 normalizes all of these annotations, and incorporates them as node-and edge-level attributes in a single semantic graph whose structure is deterministically extracted from universal dependencies (ud; @xcite syntactic parses via the predpatt tool @xcite . 1 an example graph can be seen in fig. 1 . we present the first joint uds parser, which learns to extract both graph structures and attributes from natural language input. this parser is a sequence-to-graph transductive model which takes as input a sentence and outputs a uds graph complete with node-and edge-level annotations. in contrast to the traditional semantic parsing paradigm, which shares its roots with syntactic parsing and rests on the assumption that the nodes in the graph correspond to tokens in the inputi.e. the graph is lexicalized-the parsing-astransduction paradigm treats parsing as a sequenceto-graph problem. rather than generating one sequence conditional on another sequence (sequenceto-sequence), we generate the nodes in a graph conditional on an input sequence, dynamically adding their edges during generation. as in sequenceto-sequence modeling, the supports of the input and output distributions-i.e. the input and output subspace attribute val subspace attribute val of hiller taiwan the chechnya asked bush to name leaders , , and india pakistan subspace attribute val arg. edge predicate node argument node syntax node instance edge nonhead edge syntax edge protoroles protoroles protoroles protoroles protoroles awareness change-of-location change-of-possession change-of-state existed-before … -0.110 -0.039 0.000 -0.104 1.402 factuality genericty genericity genericity time time time factual pred-dynamic pred-hypothetical pred-particular dur-days dur-minutes dur-seconds … 1.038 1.418 -0.892 1.418 -1.062 -0.912 1.260 genericity genericity genericity word-sense word-sense word-sense arg-abstract arg-kind arg-particular noun.act noun.cognition noun.food … -1.112 1.195 -1.112 -3.000 -3.000 -3.000 semantic head edge vocabularies-are not constrained to be identical. this has two benefits: first, post-hoc methods of obtaining alignments between input sequences and graphs-common especially in amr parsingare no longer required; and second, we are able to produce semantic graphs from arbitrary input vocabularies-allowing for future extensions to cross-lingual parsing @xcite . the parsing-as-transduction paradigm thus lends itself perfectly to uds parsing, since the uds protocol allows non-lexicalized (as well as cross-lingual) graphs, and these graphs may have nodes with multiple parents-i.e. re-entrant nodes-which pose problems for traditional tree-based methods but are handled natively by the transductive paradigm. we compare our end-to-end transductive parser against a strong pipeline system, finding that the parser slightly outperforms the pipeline while additionally learning to produce decompositional attribute scores. our results are reflected in the uds1.0 leaderboard at http://decomp.io/ leaderboards/ . ## related work datasets @xcite introduce the decompositional semantics framework in the context of a corpus-based verification of dowty's seminal proto-role theory of semantic roles. this work was substantially expanded by @xcite , who annotate for semantic proto-roles (spr), wordsense, and temporal properties on top of semantic graphs extracted from english web treebank (ewt; @xcite ud parses using predpatt @xcite . white et al.'s ewt annotations are modeled by @xcite , who present a crf-based multi-label classifier for proto-role labelling, and @xcite , who make use of an eventdriven neural model. more recently, the annotation coverage for the same ewt data was expanded by @xcite who annotate and model fine-grained temporal distinctions, and govindarajan et al. (2019) , who add annotations and models for genericity-i.e. the degree of generality of events and entities in linguistic expressions. all of these efforts coalesce in @xcite , which presents the first unified decompositional semantics-aligned dataset-universal decompositional semantics v1.0 (uds1.0)containing all properties annotated on top of ewt parses with standardized train, validation, and testing splits and a native reader and query interface. parsing in most work on decompositional semantics, models are tasked with learning to predict attribute values, but not the structure of the graph. @xcite develop the first model for performing both graph parsing and uds attribute prediction in a cross-lingual setting, where chinese input sentences were transduced into uds graphs derived from ud parses of the input's english translation. this represents the first application of the parsing-as-transduction paradigm to a subset of uds data as well as the introduction of a novel graph evaluation metric, s which we describe in further detail in section 5. in contrast to the end-to-end approach presented here, zhang et al. take a pipeline approach to parsing. @xcite recast semantic parsing in a tree formalism as a sequence-to-sequence problem. parsing-as-transduction, which extends this approach to directed acyclic graphs, has proven to be applicable in a variety of settings @xcite use it to achieve state-of-the-art re-sults in amr parsing. these results are improved upon and shown to generalize to two other semantic formalisms (ucca and sdp) by @xcite , which set new state-of-the-art benchmarks for amr and ucca. the former result was subsequently surpassed by @xcite , which applies a similar transductive approach, while the latter was surpassed by @xcite . having both been subjects of semeval tasks @xcite @xcite , there are a number of contrasting methods for both amr and ucca parsing. these include transition-based parsing system for amr @xcite @xcite and for ucca @xcite . in a similar vein to @xcite al. (2019b), hershcovich et al. (2018a) convert multiple formalisms into a unified formalism and use multitask learning for improved ucca parsing; however, the latter does so at a loss to performance on the other formalisms, while zhang et al. achieve state-of-the-art results in amr and ucca simultaneously. ucca has also been shown to transfer to syntactic parsing: by converting ud parse trees into a format resembling ucca, hershcovich et al. (2018b) are able to apply a ucca parser to both standard ud parses as well as enhanced ud parses, which contain re-entrant nodes. ## data the uds1.0 dataset is built on top of the ud-ewt data with three layers of annotations: ud parses, predpatt graph structure, and decompositional semantic annotations on the edge and node level. in addition to specifying the syntactic head and head relation of every token in the input, ud parses include lexical features, such as word form, word lemma, and part-of-speech (pos) tag. this forms the syntactic graph, which is lexicalized (each token is tied to a node in the graph). from these pieces of information, predpatt outputs a set of predicates and their arguments. each predicate and argument is tied via an instance edge to a particular node in the syntactic graph. because both predicates and arguments can consist of multi-word spans, there can be multiple instance edges leaving a semantic node. the semantic graph contains edges between predicates and arguments; in the case of clausal embedding, there can also be argument-argument edges. uds1.0 asked hiller bush(1) name leaders the of che. tai. ind. and pak. ## model our model is based on the transductive broadcoverage parsing model presented in @xcite , which can be consulted for further details on the encoder, decoder, and pointer-generator modules. the original parser is composed of six major modules: the encoder, the decoder embedding module, the target node module, the target label module, the head module, and the relation module. in this work we introduce two new modules: the node attribute module and the edge attribute module, as well a loss function for attributes. encoder the encoder module takes a concatenation of multiple input features: glove token embeddings @xcite , pos tag embeddings, character cnn embeddings, and bert @xcite contextual embeddings (meanpooled over subwords). these representations are passed through a stacked bidirectional lstm encoder, which has the following definition: decoder embedding module in order to generate new semantic nodes and relationships, a method of embedding categorical semantic information is required. more formally, a semantic relation is given by a tuple u i , d u i , r i , v i , d v i , where u i denotes the "head" token of index i and v i denotes the token at index i. note that these tokens are the labels of nodes in the arborescence (see fig 2 .) d u i and d v i are the indices of u i and v i , while r i is the relationship type between v i and u i . the decoder embedding module embeds these categorical variables into real space, producing a tuple of vec- for node labels u i and v i , we take the concatenation of glove and charcnn features. r i , d v i and d u i are randomly initialized. target node module from the continuous embedding of a semantic relation u let h l i and z i be defined as follows: from this last equation, we have that the generation of a new node is decomposed into three options: parsing modules to obtain a parse from the node states h 1:n , a head node and relation type must be assigned to each node 1 : n. in order to assign a head node, we instantiate two multilayer perceptrons (mlps): mlp (start) and mlp (end) , where (start) denotes the starting node of the edge and (end) denotes its target. using these mlps, for node i + 1 we obtain the next relationship r i+1 is computed in a similar fashion, also using two mlps: edge attribute module as in the case of node attributes, edge attributes do not apply in all cases. therefore, a similar bifurcation strategy is pursued with edge attribute prediction: we predict a binary attribute mask β j s,e for attribute j on edge @xmath0,e . these are given by: training the nodes in the graph are linearized in a pre-order traversal over the arborescence, which ensures that at prediction time, we have seen the potential antecendent of a node for target-side copying (e.g. bush (1) in fig. 2 ), determining the order of semantic nodes in the graph. the syntactic children of these nodes are presented in the order they appear in the text. the loss functions for the node, head, and relation prediction modules are cross-entropy loss, while for the masks α and β binary cross-entropy loss is used, since each position in the mask is a separate classification decision. the loss function used for k attributes ν 1:k on n nodes/edges is given by: this loss encourages the predicted attribute ν j i value to be close in value to the true value ν j * i via the mean-squared error criterion while concomitantly encouraging the predicted and reference values to share a sign via the thresholded cross-entropy criterion. both node and edge attribute models are trained to predict attribute values independently, and that parameters are shared across attributes. this is central to our analysis in §7. following @xcite we train the structural parsing modules with coverage loss @xcite . all models were trained to convergence using the adam optimizer (kingma and ba, 2014) with a learning rate of 0.001. ## experiments pipeline model recall from section 3 that the semantic graph structure in uds graphs is deterministically generated from predpatt, which takes as input a ud parse and outputs a semantic graph structure. this leads to a strong pipeline model for the graph structure alone: running a high-performing ud parser-the stanford ud parser (chen and manning, 2014)-and passing its output through predpatt to create a structure. the only source of error is the ud parsing model, which for english performs very highly. s metric for evaluating the quality of output graph structures, smatch @xcite , a hill-climbing approach to approximating the optimal matching between variables in two graphs, is commonly used. while smatch can match categorial variables such as those found in meaning representations like amr, it lacks a matching function for continuous variables such as decompositional attributes. to remedy this, @xcite introduce the s metric, an extension to smatch that allows for attribute matching. using hill-climbing, we are able to match instance and attribute nodes and edges; instance nodes are matched via string match, while attribute similarity is given by 1 - ## results the baseline column in ## analysis note that the numbers reported in tables 2 and 4 are not directly comparable, as the scores in in examining instances with an s score below 50, we observe two trends: the input sentences are often ungrammatical, and for 63.82% (on the validation set) the model predicts no output nodes. while the pipeline system does well on modeling semantic graph structure, it is by its definition unable to perform attribute parsing. in contrast, the results presented in tables 4 and 5 show that the parser can jointly learn to produce semantic graphs and annotate them with attributes. finally, we find that while weighting the loss with the confidence scores has a small benefit in the semantics-only setting, it hurts overall attribute and structure prediction performance. this may be due to the relatively small size of the uds dataset, which makes a strategy that is effectively weakening the loss signal at training time less effective. 6 figs. 3a-3c show the correlational strength coefficient between the true and predicted attributes under a forced decode of the graph structure. it is defined over property types indices j, k with predicted attribute values ν j i and true values ν j * i as: fig. 3a shows the model typically systematically under-or over-predicts the values for pairs sentences property (a) ours (a) (b) ours (b) (c) ours (c) as well as between genericity-arg-abstract, which rates how conceptually abstract an argument is, and the cognition wordsense, which applies to abstract terms such as "doubts" and "thoughts". in fig. 3b , we again observe several negative ψ values; however, some positive correlations can be seen between certain time properties, such as duration-days, duration-weeks, and durationmonths, as well as more strongly positive ψ's between certain genericity annotations. the positive ψ between factuality and genericity-hypothetical indicates the model has captured the commonalities between predicates with these annotations. in contrast to the node attributes, fig. 3c shows stronger results for edge attribute prediction, with all significant ψ's being positive, and related attributes falling into clusters (e.g. volition, awareness, sentience, or the existed attributes) qualitative examples ## conclusion the scalar valued, multi-attribute nature of uds provides for a distinct structured prediction problem as compared to other existing representations. we have demonstrated how a transductive parsing paradigm that has achieved state-of-the-art results on other representations can be adapted to uds1.0 structures and attributes, and have provided procedures for analysis, with the fine-grained nature of uds allowing for investigating novel correlations and aspects of meaning. while uds structures and various attribute types have been modeled separately @xcite @xcite @xcite , this work represents the first time all of these attributes and structures have been modeled jointly, and establishes a baseline for future efforts on uds1.0. we envision future efforts exploring the interactions between improving the underlying graphstructure prediction and ever-better correlations to human judgements on individual properties.