Auxiliary tasks to boost Biaffine Semantic Dependency Parsing (2022)

Abstract: The biaffine parser of (CITATION) was successfully extended to semantic dependency parsing (SDP) (CITATION). Its performance on graphs is surprisingly high given that, without the constraint of producing a tree, all arcs for a given sentence are predicted independently from each other (modulo a shared representation of tokens).To circumvent such an independence of decision, while retaining the O(n^2) complexity and highly parallelizable architecture, we propose to use simple auxiliary tasks that introduce some form of interdependence between arcs. Experiments on the three English acyclic datasets of SemEval-2015 task 18 (CITATION), and on French deep syntactic cyclic graphs (CITATION) show modest but systematic performance gains on a near-state-of-the-art baseline using transformer-based contextualized representations. This provides a simple and robust method to boost SDP performance.

## introduction and related work semantic dependency parsing is the task of producing a dependency graph for a sentence. depending on the datasets, these dependencies may correspond to predicate-argument relations, with labels numbering semantic arguments (as in figure 1-top) or dependencies with intermediate status between syntax and semantics, with labels being canonical grammatical functions that normalize syntactic alternations (e.g. in figure 1 -bottom, the clitic l' (him) is the canonical object of the passive verb form sollicité (solicited)). if one views each dependency as a decision to make, both dependency parsing (dp, outputing a syntactic tree) and semantic dependency parsing (sdp) are known to exhibit high interdependence of decisions. for instance in dp, when parsing a question answering machine, choosing machine as root is linguistically coherent with the machine she went back and spoke to the desk clerk. cela l' a habitué à être très sollicité. this him has accustomed to be very solicited. suj obj mod obj figure 1 : top: english semantic graph in the dm format, as part of the semeval2015-task18 dataset @xcite . bottom: french deep syntactic graph as defined by @xcite . in dp though, the interdependence between arcs is partially solved by the tree constraint: choosing one head for a given token amounts to ruling out all other heads. this structural interdependence is absent in sdp. complex structural, lexical and semantic factors control whether a given dependent should be attached to zero, one or several heads. several approaches exist in the literature to capture interdependence of arcs in sdp, which often derive from proposals made for dp. one is to use a higher-order graph-based parser. @xcite achieve state-of-the-art results (without pretrained lm) on the english part of the semeval2015-task18 data, by using second-order factors to score the graphs, yet at the cost of a o(n 3 ) complexity. another main approach is to use sequential decisions, and hence take advantage of previous decisions by encoding the previously predicted arcs. this is the case in the transition-based parser of @xcite , or in the system of @xcite , which selects a new head for certain tokens at each iteration, using reinforcement learning to order this selection of heads. both models have a o(n foot_0 ) complexity (when used without cycle detection), and in both cases, sequential decisions benefit from the encoding of previously predicted arcs, yet at the cost of error propagation. for that reason, @xcite propose a system close to that of @xcite , yet allowing the system to overwrite previous decisions and hopefully correct itself. on the contrary, the biaffine system of @xcite (hereafter dm18) performs a simultaneous scoring of all candidate arcs, decides to predict an arc independently of the other ones. this results in a highly parallelizable o(n 2 ) inference, with surprisingly high performance albeit below second-order parsing. as for most nlp tasks, sdp performance increases when integrating transformer-based contextual representations when encoding input tokens. on the english dataset from the semeval 2015 task 18 @xcite , fernández-gonzález and gómez-rodríguez (2020) (hereafter fg20) report a +0.7 and +2.0 increase for the in-domain (id) and out-of-domain (ood) test sets respectively. 1 in this work, we retain the simple o(n 2 ) biaffine architecture of dm18, and we investigate how simple auxiliary tasks can introduce some interdependence between arc decisions, in a multi-task learning setting @xcite . we show modest but statistically significant improvements on the three english datasets of the widely used semeval2015-task18 data @xcite . we also test another appealing property of the biaffine architecture, which is the absence of formal constraints on the output graphs. experiments on french deep syntactic graphs @xcite , which are highly cyclic, also demonstrate the effectiveness of our auxiliary tasks for sdp. 1 using the biaffine dm18 architecture, he and choi (2020) report a +2 and +3 point increase in id and ood. yet, these results are not comparable: the authors have used a different pre-processing, which adds orphan dependencies from the root to orphan tokens, resulting in an easier task (p.c. with the authors and https://github.com/emorynlp/bert-2019/is sues/1 ). ## the baseline biafine graph parser we reuse the computation of the arc and label scores of the dm18 model, which we modernized by using contextual representations: input sequence w 1:n is passed into a pretrained language model. we represent a word-token w i by concatening the contextual vector of its first subword 2 h (bert) i and a word embedding e (word) i . for some of the experiments, we also concatenate a lemma and a pos embedding. the sequence of word-tokens representations is passed into several bilstm layers: r 1:@xmath0:n ). the recurrent representation r i is then specialized according to two binary features: head versus dependent, and arc versus label score: we use a simplified biaffine transformation for arc scores, and a per-label one for label scores: ## auxiliary tasks targeting sets of arcs preliminary experiments on english semantic dependency graphs @xcite and on french deep syntactic graphs @xcite have shown us that the biaffine graph parser gives good results, but with inconsistencies that are clearly related to the locality of decisions. in particular, a quick error analysis revealed incompatible here is a good rule of thumb : ... arc combinations. more precisely, we noticed impossible sets of labels for the set of heads of a given dependent. for example in the dm part of the semeval2015-task18 dataset, tokens are sometimes attached with a mwe label (for a component of a multi-word expression) and attached to another head with a non-mwe label, as shown for the rule token in figure 2 . this incorrect situation actually never happens in the training set, but this impossibility is not captured by the model. in the predicted french deep syntactic graphs, we noticed punctuation tokens wrongly attached to two different heads with the specific punct dependency label. a second observation is a tendency in some of the datasets to predict disconnected tokens (i.e. with no incoming nor outgoing arcs) too frequently. more generally, when counting the number of predicted heads for each token in the predicted graphs, the accuracy is about 95% in the english datasets, and below 92% for the french one. ## conclusion when using a biaffine graph-based architecture for semantic dependency parsing (sdp), arcs are predicted independently from each other. our contribution is a set of simple yet original auxiliary tasks that introduce some form of interdependence of arc decisions. we showed that training recurrent word-token representations both for the sdp task and for predicting the number of heads and the incoming labels of each word is systematically beneficial, when tested either on english or french, on semantic or on deep syntactic graphs, and on inor out-of-domain data. demonstrations, pages 38-45, online. association for computational linguistics. yuan zhang and david weiss. 2016. stack-propagation: improved representation learning for syntax. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1557-1566, berlin, germany. association for computational linguistics.