Multiple Tasks Integration: Tagging, Syntactic and Semantic Parsing as a Single Task (2021)

Abstract: Departing from both sequential pipelines and monotask systems, we propose Multiple Tasks Integration (MTI), a multitask paradigm orthogonal to weight sharing. The essence of MTI is to process the input iteratively but concurrently at multiple levels of analysis, where each decision is based on all of the structures that are already inferred and free from usual ordering constraints. We illustrate MTI with a system that performs part-of-speech tagging, syntactic dependency parsing and semantic dependency parsing. We observe that both the use of reinforcement learning and the release from sequential constraints are beneficial to the quality of the syntactic and semantic parses. We also observe that our model adopts an easy-first strategy that consists, on average, of predicting shorter dependencies before longer ones, but that syntax is not always tackled before semantics.

## introduction historically, natural language processing (nlp) systems have generally been built as sequential pipelines, where each module adds another layer of annotation, in order of (supposed) increasing complexity. progress in neural networks has, however, led to the development of state-of-the-art systems that completely bypass intermediate levels of analysis that were previously considered essential. for example, the system of @xcite perform semantic role labeling (srl; @xcite without referring to any explicit (morpho)syntactic information. a well known problem of sequential systems is error propagation, which happens when an incorrect prediction at some point in the process leads to more incorrect predictions at a later stage. in traditional pipelines, one of the roots of error propaga-tion lies in the fact that they feature a unidirectional flow of annotation between their different stages. end-to-end systems with no intermediate level of analysis, in contrast, are protected against one form of error propagation. since such systems are not contingent on possibly faulty levels of analysis, they are free from the interference they could cause. however, the absence of any intermediate decisions -symbolic traces of the system's computation -raises questions about generalisation ability and interpretability @xcite . what is more, by reducing the information in the prediction and training signals at their disposal, they might not leverage the full power of the neural networks they are usually based on. indeed, in their reintroduction of syntactic dependencies to the training process, @xcite managed to develop a state-of-the-art system for srl that at the same time computes good-quality syntactic parses. departing from these two kinds of architectures, we propose multiple tasks integration (mti). the principles of mti are (i) to let the system take actions pertaining to different levels of analysis without constraining their order, and (ii), at any given point of the process, to feed all layers of annotation as input for the next predictions. the different tasks can therefore fully interact with each other as if they were a single task. our main contribution lies in an illustration of these principles with a system that performs part-of-speech (pos) tagging, syntactic dependency parsing and semantic dependency parsing (sdp), on english data. we have chosen these specific tasks not only for their strong interdependence but also for their generality: many other tasks in nlp (e.g. srl, coreference resolution, relation extraction) can be reduced to labelling or bi-lexical dependencies creation problems. we show that in this specific case, letting the system order freely its actions across all three tasks leads to better performance, and that this im-provement concerns the syntactic layer as well as the semantic one, although to a lesser extent. since its apparition for speech recognition @xcite , beam search has become ubiquitous in nlp and artificial intelligence. exploring multiple paths of actions in parallel instead of a single one mitigates the risk of error propagation of both kinds. an orthogonal approach consists in designing systems able to attain the gold annotation via multiple paths (usually involving different orderings of subsequences of actions) and training them to infer an easy-first strategy. such a strategy corresponds to performing easy actions first so as to maximise the amount of reliable information available when having to tackle harder decisions. this idea has been applied to various nlp problems @xcite @xcite , but, to the extent of our knowledge, only in single-task contexts. another promising research direction is the one of iterative refinement. the idea here is to use a baseline system in order to produce an initial output and to then correct it, possibly multiple times, with a refinement system that has access to both the previous output and the input of the problem. iterative refinement techniques have been used for tasks such as machine translation @xcite and srl @xcite . other techniques are designed to help to fight error propagation of the second kind, which focus on the training of the system to make it robust to its own mistakes. one possibility consists in training the system to predict the next best action in any state it find itself in, instead of simply staying on the (errorless) gold path. doing so requires the possibility for a next best action to be determined, which, while trivial in some cases such as pos tagging, is not in others, in particular for structured prediction. hence the introduction of the notion of dynamic oracle by @xcite for syntactic parsing. an alternative is reinforcement learning (rl; @xcite , which also trains the system from its actual trajectories, but depends instead on rewards defined for each action in the particular state it is taken. note that in adequate settings, rl has been shown to lead to the emergence of easy-first strategies. this is the case in semantic parsing, as shown in the work of kurita and søgaard (2019), which we extend here to both pos tagging and syntactic parsing following our proposed multiple tasks integration principles rather than implementing a traditional multitask learning scheme. ## experiments to test whether mti is a viable paradigm and determine the impact of rl, in this section we test the model described above along with three variants. these four models can be seen as combinations of two binary traits. the first trait pertains to the ordering of the actions: sequential models simulate sequential pipelines, while for free models, as above, no particular constraint is imposed. a sequential model can only select tag-t actions during the first step of an episode, only syn-j-l and root during the second and only sem-j-l, top pred and halt afterwards. this is ensured by using as policy for each token the vector obtained by normalising a masked version of its usual policy. 9 the second trait pertains to the learning process: supervised models are only trained in the pre-training phase while rl models, as above, are further trained using reinforcement learning. ## conclusion and future work we have defined multiple tasks integration as a set of principles for joint processing, orthogonal to weight sharing. the essence of mti is to process the input iteratively but concurrently on multiple levels of analysis, basing each decision on all of the structures already inferred and free from usual ordering constraints. this way, the different tasks can interact in the full sense of the term. to train such a system, we propose using reinforcement learning algorithms, thus allowing it to infer its own ordering strategy. in practice, we have trained a system to perform part-of-speech tagging, syntactic dependency parsing and semantic dependency parsing. we have observed that both the use of reinforcement learning and the release from sequential constraints are beneficial, not only to the (seemingly) highest level task (i.e. semantic parsing), but also to some intermediate ones (i.e. syntactic parsing). if the inferred strategies are interpreted as being easy-first -which is supported by the fact that shorter dependencies have a strong tendency to be generated before longer ones -, then we have observed that syntactic parsing is not necessarily simpler than semantic parsing and that both benefit from being executed concurrently. while our model is not yet as effective as today's most complex systems, it is still competitive with most of the parsers presented in the recent literature, even though it uses a poorer input signal for inference (consisting of the raw tokens only). furthermore, several aspects of the current system are open to developments that seem likely to improve performance. for instance, we do not use here the full potential of reinforcement learning, as what we optimise (the expected sum of rewards) is not the metric we are interested in (which would be either an average of the three f1 or the semantic one). for each of the three tasks, the sum of the rewards we have defined approaches, up to a multiplicative constant, the corresponding f1 of the system when the latter approaches 100%, but a better approximation might prove more successful. in a similar vein, kurita and søgaard (2019), who work with a similar architecture as far as sdp is concerned, penalise halt actions when relevant dependencies are still missing, which intuitively boosts recall. note also that our network is still rather simple, in that it does not use any form of regularisation nor any advanced technique to handle out-of-vocabulary words. we would also be interested in applying mti to other tasks and in studying how well it can learn from incomplete annotations. our architecture can be straightforwardly adapted to any labelling or graph building task, as long as all nodes are tokens of the input sentence. in contrast, work remains to be done in order to handle formalisms such as abstract meaning representation (amr, @xcite or on how to integrate a generation component with, for example, the goal of translating the sentence being analysed. yu zhang and qiang yang. 2018. a survey on multi-task learning. arxiv:1707.08114 [cs]. jie zhou and wei xu. 2015. end-to-end learning of semantic role labeling using recurrent neural networks. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1127-1137, beijing, china. association for computational linguistics.