Joint Universal Syntactic and Semantic Parsing (2021)

Abstract: While numerous attempts have been made to jointly parse syntax and semantics, high performance in one domain typically comes at the price of performance in the other. This trade-off contradicts the large body of research focusing on the rich interactions at the syntax–semantics interface. We explore multiple model architectures that allow us to exploit the rich syntactic and semantic annotations contained in the Universal Decompositional Semantics (UDS) dataset, jointly parsing Universal Dependencies and UDS to obtain state-of-the-art results in both formalisms. We analyze the behavior of a joint model of syntax and semantics, finding patterns supported by linguistic theory at the syntax–semantics interface. We then investigate to what degree joint modeling generalizes to a multilingual setting, where we find similar trends across 8 languages.

## introduction given their natural expression in terms of hierarchical structures and their well-studied interactions, syntax and semantics have long been treated as parsing tasks, both independently and jointly. one would expect joint models to outperform separate or pipelined ones; however, many previous attempts have yielded mixed results, finding that while one level can be used as an additional signal to benefit the other @xcite @xcite , obtaining high performance in both syntax and semantics simultaneously is difficult @xcite . a variety of tree-and graph-based representations have been devised for representing syntactic structure (e.g., varieties of constituency and dependency parse trees) as well as semantic structure, for example, abstract meaning representation (amr; @xcite , universal conceptual cognitive annotation (ucca; @xcite , and semantic dependency parsing formalisms (sdp; @xcite . these semantic representations have varying degrees of abstraction from the input and syntax, ranging from being directly tied to the input tokens (e.g., sdp formalisms) to being heavily abstracted away from it (e.g., amr, ucca). universal decompositional semantics (uds; @xcite ) falls between these extremes, with a semantic graph that is closely tied to the syntax while not being constrained to match the input tokens. crucially, uds graphs not only represent the predicate-argument relationships in the input, but also host scalar-valued crowdsourced annotations encoding a variety of semantic inferences, described in §3. these provide another level for linguistic analysis and make uds unique among meaning representations. furthermore, as uds graphs build on universal dependency (ud) parses, uds is naturally positioned to take full advantage of the extensive and linguistically diverse set of ud annotations. we extend the transductive sequence-to-graph uds parser proposed by @xcite to simultaneously perform ud and uds parsing, finding that joint modeling offers concomitant benefits to both tasks. in particular, after exploring several multitask learning objectives and model architectures for integrating syntactic and semantic information, we obtain our best overall results with an encoder-decoder semantic parsing model, where the encoder is shared with a biaffine syntactic parser @xcite . because the uds dataset is annotated on an existing ud corpus, our experiments isolate the effect of adding a semantic signal without the confound of additional data: all our monolingual systems are trained on the same set of sentences. in contrast to previous work on joint syntaxsemantics parsing, we are able to achieve high performance in both domains with a single, unified model; this is particularly salient to uds parsing, as the ud parse is a central part of a complete uds analysis. our best joint model's uds performance beats the previous best by a large margin, while also yielding sota scores on a semantically valid subset of english web treebank (ewt). we introduce a model optimized for ud, which can obtain competitive uds performance while matching the current sota ud parser on the whole of ewt. 1 we analyze these objectives and architectures with lstm encoders and decoders as well as novel transformer-based sequence-tograph architectures, which outperform the lstm variants. while previous work suggests that contextualized encoders largely obviate the need for an explicit syntactic signal in semantic tasks @xcite , we find that syntactic (and semantic) annotations provide consistent performance gains even when such encoders are used. this suggests that the ud and uds signals are complementary to the signal encoded by a pretrained encoder, and so we tune the encoder at various depths, further improving performance. building on this result, we leverage the shared multilingual representation space of xlm-r @xcite to examine ud parsing in 8 languages across 5 families and varying typological settings, where we demonstrate a cross-lingual benefit of uds parsing on ud parsing. ## background and related work in both language production and comprehension, syntax and semantics play complementary roles. their close relationship has also been noted in language acquisition research, with the ''semantic bootstrapping'' hypothesis proposing that infants use semantic role information as an inductive bias for acquiring syntax @xcite ), foot_1 while @xcite , @xcite , and naigles (1990) present evidence that infants use syntactic information when acquiring novel word meanings. their connection was codified in @xcite seminal work formalizing the link between syntactic structures and formal semantics. broadly, their interactions can be split into ''bottom-up'' constraints on the semantics of an utterance from its syntax, and ''top-down'' constraints on the syntax, based on the semantics. despite their close empirical and theoretical ties, work on predicting syntax and semantic structures jointly has often struggled to attain high performance in one domain without compromising on performance in the other. ## data a number of factors make the uds representation @xcite particularly well-suited to our purposes, especially the existence of parallel manually annotated syntactic and semantic data. in uds, a semantic graph is built on top of existing ewt @xcite ud parses, which are mapped to nodes and edges in a semantic dag via a set of deterministic rules @xcite . this semantic graph, which represents the predicate-argument relationships in the text, is then annotated with crowdsourced scalar-valued attributes falling into the following categories: factuality (how likely a predicate is to have occurred), genericity (how general or specific a predicate/argument is), time (how long an event took), word sense (which word senses apply to a predicate/argument), and semantic proto-roles, which break the traditional srl ontologies into simpler ''proto-agent'' properties (e.g., volition, awareness, sentience) and ''proto-patient'' properties (e.g., change of state, change of location, being used). note that while the semantic graph structure is tied to the syntax, the attribute values, encoding fine-grained, abstracted semantic inferences, are not. these attributes are unique among graph-based semantic representations. all of these properties are annotated on a scale of -3 to 3; for more details on the dataset, we refer the reader to @xcite and @xcite , as well as figure 1 . we train and evaluate on a semantically valid subset of ewt. 3 we similarly limit our baselines to these examples for our ud analysis, and release our cleaned ud dataset, for which we report state-of-the-art parsing performance. arborescence following @xcite and @xcite , we convert the uds graph into an arborescence, or tree. reentrant semantic nodes are copied and co-indexed, so that a dag can be recovered via a deterministic post-processing step. each node is assigned a token label, taken from the token corresponding to the syntactic head of the semantic node (instance edges in figure 1 ). all syntactic nodes not assigned as labels to semantic nodes are included as nodes dominated by their corresponding semantic node (pink edges in figure 2 ). 4 an example conversion of the uds graph in figure 1 can be found in figure 2 . in the ''semantics only'' setting, only the semantic nodes are included in the graph. a pre-order traversal linearizes the arborescence into a sequence of nodes, indices, edge heads and labels, and the corresponding node and edge attributes. ## models we build on the transductive parsing model presented by @xcite , which itself builds on the broad-coverage semantic parsing model of @xcite and relies heavily on allennlp @xcite . the transductive parsing paradigm recasts graphbased parsing as a sequence-to-graph problem, using an attentional sequence-to-sequence model to transduce the input sentence into a set of nodes while incrementally predicting edge and edge labels for those nodes. the uds semantic parser consists of 7 modules: the encoder module embeds the input features (type-level glove and contextualized word embeddings, pos tags, charcnn features) into a latent space, producing one vector per input token. bert representations are pooled over subword units. 5 the decoder embedding module embeds the categorical information of the previous timestep (e.g., the token identity and index, the head token identity and index, the edge type) into a real space. the target node module builds node representations from the decoder embedding module's output. the target label module extends the pointer-generator network @xcite , which supports both generating new token labels from a 5 this input is augmented with a sinusoidal position embedding for the transformer models. the relation module is a graph-based dependency parser based on the parser presented by @xcite which uses separate head and dependency mlps followed by a biaffine transformation to predict the dependency scores and labels between each node in a fully connected graph. for semantic parsing, we follow a greedy decoding strategy since the linearization of the arborescence implicitly enforces a well-formed output; this allows for single-step online decoding. the node attribute module uses the node representations to predict whether each attribute applies to each node, and what its value should be. deciding whether the attribute applies and the prediction of its value are performed by separate mlps. the edge attribute module is similar to the node attribute module, but passes a bilinear transformation of two node representations to the mlps, which predict edge-level properties and masks. in this work, we make several modifications to this model. firstly, we replace the encoder and target node modules with transformer-based architectures @xcite . next, we introduce a second biaffine parser on top of the output of the encoder module, which is tasked with performing dependency parsing on the ud data. during training, we use greedy decoding, while at test time the chu-liu-edmonds maximum spanning tree algorithm @xcite is used. more specifically, the encoded input representations s t in stengel-eskin et al. (2020) were obtained from a stacked bidirectional lstm. while the input to this module remains the same in our transformer-based implementation, the representation s t is now given by the final layer of a multi-head multi-layer transformer model. crucially, following @xcite we replace the layer normalization layer with a scalenorm layer and place it before the feedforward network. the syntactic biaffine parser uses the encoder representations s to predict a head and head label for each token. this task is different from the semantic parsing task as the graph is lexicalized (i.e., there is a bijection between input tokens and graph nodes). similarly, the node representations z i are computed by a transformer decoder with both self-attention (as in the encoder) and source-side attention. the first layer of node representations for the decoder is given by learned continuous embeddings of the head token and current token representation, their respective indices, and the relationship between them. during training, the gold nodes and heads are used (i.e., teacher forcing), and the attention is computed with an autoregressive mask, so that each token is only able to attend to tokens in its left context. we take z @xmath0, x l i being the output of the final layer in a stack of l transformer layers. we also follow nguyen and salazar in scaling the attention head weight initialization by a factor of k. ## experiment 1: joint english parsing to determine the effect of jointly parsing the syntax and semantics, we consider a number of baselines and experimental settings. first, we contrast our re-implementation of stengel-eskin et al.'s (2020) lstm-based model with their results. we then report the results of our transformer-based model, described in §4. after establishing these baselines, we consider different methods of incorporating the syntactic signal into the model: concat-before (cb): here, we linearize the syntactic ud parse, which is a tree, via a pre-order traversal, yielding a sequence of nodes, edge heads, and edge labels, which we prepend to the corresponding sequences obtained by linearizing the semantic parse, separated by a special separation token. at inference time, we use this token to split the output into syntactic and semantic parses. concat-after (ca): this setting is identical to the concat-before setting, except that the syntactic graph is appended at the end of the semantic sequence. these two settings incorporate some syntactic signal into the semantic parse, but do not exploit ud parsing's lexicalization assumption; thus we expect them to yield subpar ud results. ## experiment 1: results and analysis the transformer outperforms the lstm on uds parsing. we first observe that, with modifications and tuning, the transformer architecture strictly outperforms the lstm despite the relatively low number of training examples (12.5k). figure 3 , which corresponds to joint training has little impact on attribute metrics for non-baseline models. figure 5 shows pearson's ρ and binarized attribute f1 (with θ tuned on the development set); this corresponds to the second two rows and final 10 rows of we see that both for the lstm and the transformer, the encoder-side model has about the same performance for ρ and attribute f1 as the uds-only model, and the transformer variants consistently out-perform their lstm counterparts. in contrast, the addition of syntactic information through concatenation (cb, ca) seems to diminish the performance on these metrics. for the lstm, the intermediate model has lower performance than the encoder-side variant, while for the transformer it is almost identical. joint parsing slightly improves semantic structural performance. figure 6 shows the structural f1 computed by s score, where we observe that the lstm's performance, which is lower than the transformer's in the baseline setting, benefits most from the concatenation settings, while suffering under the encoder and intermediate settings. by way of contrast, the transformer, whose baseline performance is higher, benefits most from the encoder-side biaffine parsing setting, which also boasts the best ud performance (cf. figure 4 ). while the concat-after setting offers s score improvements for both encoder/decoder types, the syntactic performance in this setting is very poor (< 60 uas). the transformer encoder-side multitask model is able to improve structural performance for the encoder-side while simultaneously boosting uas and las (see figure 4 ). these results demonstrate that explicitly incorporating a syntactic signal into a transductive semantic parsing model can be done without damaging semantic performance, both for uds attributes and structure. perhaps more surprisingly, the semantic signal coming from the uds attributes and structure improves the syntactic performance of the model when the syntactic model is able to take advantage of the lexicalized nature of ud. note that due to the parallel nature of the ud and uds data, we can conclude that the improvements here result from the additional structural signal, and not merely from the addition of more sentences. we see that for the concatenation settings, while the semantic structural performance may increase, the syntactic parsing results are dismal. this is true whether we concatenate the syntactic graph before or after the semantic one. this may be explained by the fact that, by using a transductive model for a lexicalized parsing task like uds, we are complicating what the model needs to learn. rather than simply labeling existing nodes, the model must reproduce these nodes via source-side copying. while prima facie, we would expect the intermediate model to outperform the multitask encoder-side model, as its decoder has explicit access to the syntactic parse, we see that this is not the case; it shows lower structural and attribute performance. this represents a direction for future work. the role of the encoder the results in figure 5 show that the transformer-based model has a heavy advantage over the lstm-based model in terms of attribute prediction. this might be due to an improved ability by the transformer encoder to incorporate signals from across the input, since the self-attention mechanism has equal access to all positions, while the bilstm has only sequential access which may become corrupted or washed out over longer distances. given the highly contextual nature of uds inferences, we would expect a model that better captures context to have a distinct advantage, as the crucial tokens for correctly inferring an attribute value may be found in a distant part of the input sentence: for example, if we wish to infer the factuality of ''left'' in the sentence ''bill eventually confessed to the officers that, contrary to his previous statements, joan had left the party early,'' most of the signal would come from the token ''confessed,'' producing a high score. the construction of uds and its attributes' scalar nature allow us to test this hypothesis by examining the pearson correlation between predicted and true attributes at different positions in the input. 7 in order to compare the correlations across sentences, we group the predicted and reference attributes into 10 percentile ranges, based on the ratio of the node position and the sentence length, that is, what percentage into the sentence the node occurs. we then average all pearson ρ values across all attribute types in each bin, obtaining average ρ values by sentence completion percentile. with this data, we can compare two models on a very fine-grained level, asking 7 we use the syntactic head nodes of each semantic node to propagate positional information to the semantic nodes. questions such as, ''how much better does the transformer model do than the bilstm on nodes that are between 0% and 10% into the sentence?'' figure 7 shows such comparisons between a unidirectional left-to-right lstm encoder and the bidirectional lstm, and between the bidirectional lstm and the transformer. while the unidirectional lstm actually outperforms the bilstm in the central percentiles, it struggles near the edges. this could be explained on the left edge by a lack of right context, and on the right edge by difficulties with long-range dependencies. furthermore, the transformer outperforms the bilstm at all positional percentiles, but particularly in the central regions, suggesting that, while the bilstm is able to incorporate contextual information well at the edges of a sentence, the information is diluted in the central region, while the transformer's self-attention mechanism is equally able to draw from arbitrary positions at all timesteps. if indeed the combination of uds and ud signals provides information not yet encoded in bert, then fine-tuning bert with these signals should yield additional benefits. following observations that syntax and semantics are encoded to varying degrees at different depths in contextualized encoders @xcite @xcite with syntactic information typically lower in the network, we explore the trade-off between freezing and tuning various layers of the bert encoder. specifically, we tune the top n layers, starting from a completely frozen encoder and moving to tuning all 12 layers. 8 intuitively, one might expect to see a monotonic increase as the number of tuned layers increases, as each additional unfrozen layer provides the model with more capacity. however, the results presented in figure 8 show a more nuanced trend: while the performance across syntactic and semantic metrics increases up to a point, they begin to decrease again when additional layers are unfrozen. this may be due to data sparsity; given the relatively small size of the uds corpus, the addition of too many parameters may encourage overfitting, resulting in decreased test performance. note that the encoder-side model all three panels of figure 8 is the same model, that is, the best uas, las, and s score performance is obtained by the same model, and that the performance of the joint model at any given tuning depth typically falls above that of the baseline. ## experiment 3: multilingual parsing the results in §6 that english ud parsing and uds parsing are mutually beneficial naturally give rise to a follow-up question: does this relationship extend to a multilingual setting. as in the monolingual case, we explore both the impact of ud parsing on uds, and vice versa. ud is by design highly multilingual, spanning scores of languages from a diverse typological range, often with multiple treebanks per language. this has led to interest in evaluating the performance of ud parsing models not just on english, but across a range of languages and language families; both the 2017 and 2018 conll shared tasks focused on multilingual ud parsing @xcite . the introduction of multilingual contextualized encoders, such as mbert @xcite and xlm-r @xcite has enabled models to perform ud parsing in multiple languages simultaneously by using features obtained by a single multilingual encoder @xcite . by initializing weights for one task (syntactic or semantic) with weights learned on the other, and leveraging the shared input representation space of xlm-r, we examine bottom-up effects of representations learned with a syntactic objective languages eight languages in 5 families from the 2018 conll shared task (zeman et al., 2018) were chosen, across both higher and lower resource settings. the trends here suggest that a multilingual syntactic signal, when incorporated well into a uds model, can provide benefit to the syntactic performance without necessarily reducing the semantic performance. note that unlike in §6, the syntactic data used to pretrain the syntactic encoder and biaffine parser is neither parallel to the uds dataset, nor is it in english. thus, that the syntactic data can act as a signal for semantic parsing, albeit with small effects, is surprising. ## conclusion in §5, we introduced a number of multitask architectures for joint syntactic and semantic parsing, which we demonstrated in §6 can perform ud and uds parsing simultaneously without sacrificing performance, as evaluated across a number of syntactic and semantic metrics. in particular, we observed a top-down benefit to syntactic parsing from the semantic signal as well as a bottom-up benefit to semantic performance from syntactic parsing. we contrasted both lstm and transformer-based variants of these architectures, finding the transformer to be better on all metrics. finding the syntactic and semantic information present in the data to be complementary to that encoded in a frozen contextualized encoder, we experimented in §7 with tuning the encoder to varying depths, finding that tuning the top-most layers provides the greatest benefit. we analyzed the models resulting from this tuning step on their ability to resolve attachment ambiguities, as well as examining interactions between uds annotations and ud dependency relations. furthermore, in §8, we extended our experiments beyond english, using a transfer-learning experimental paradigm to investigate effects between multilingual syntactic parsing in 8 languages and english semantic parsing, where we found similar trends to the englishonly setting. based on these multilingual results, we believe that expanding the uds data paradigm (i.e., ud-based graph structure, continuous attributes) beyond english and building robust multilingual parsing models is a particularly promising direction for future work. other directions include improving the robustness of the transformer model in low-resource settings and improvements to the attribute modules.