Multitask Easy-First Dependency Parsing: Exploiting Complementarities of Different Dependency Representations (2020)

Abstract: In this paper we present a parsing model for projective dependency trees which takes advantage of the existence of complementary dependency annotations which is the case in Arabic, with the availability of CATiB and UD treebanks. Our system performs syntactic parsing according to both annotation types jointly as a sequence of arc-creating operations, and partially created trees for one annotation are also available to the other as features for the score function. This method gives error reduction of 9.9% on CATiB and 6.1% on UD compared to a strong baseline, and ablation tests show that the main contribution of this reduction is given by sharing tree representation between tasks, and not simply sharing BiLSTM layers as is often performed in NLP multitask systems.

## introduction dependency parsing is the task of assigning a syntactic structure to a sentence by linking its words with binary asymmetrical typed relations. in addition to syntactic information, dependency representations encode some semantic aspects of the sentence which make them important to downstream applications including sentiment analysis @xcite and information extraction @xcite . in this paper we are interested in arabic dependency parsing for which two formalisms have been developed (see §2). the first is the columbia arabic treebank (catib) representation @xcite , which is inspired by arabic traditional grammar and which focus on modeling syntactic and morpho-syntactic agreement and case assignment. the second is the universal dependency (ud) representation @xcite , which has relatively more focus on semantic/thematic relations, and which is coordinated in design with a number of other languages @xcite . while previous work on arabic dependency parsing @xcite tackled these formalisms separately, we argue that they stand to benefit from multitask learning (mtl) @xcite . mtl allows for more training data to be exploited while benefiting from the structural or statistical similarities between the tasks. we therefore propose to learn catib and ud dependency trees jointly on the same input sentences using parallel treebanks. deep neural networks are particularly suited for multitask scenarios via straightforward parameter and representation sharing. some hidden layers can be shared across all tasks while output layers are kept separate. in fact, most deep learning architectures for language processing start with sequential encoding components such as bilstms or transformer layers which can be readily shared across multiple tasks. this approach is widely applicable even to tasks that use very different formalisms and do not have parallel annotations @xcite . this type of sharing has also been shown to benefit (semantic and syntactic) dependency parsing, both transition-based @xcite and graph-based @xcite . in addition to simple parameter sharing, joint inference across multiple tasks has been shown to be beneficial. @xcite perform decoding jointly across multiple semantic dependency formalisms with cross-task factors that score combinations of substructures from each @xcite . joint inference however comes with increased computational cost. to address this issue, we introduce a multitask learning model for dependency parsing ( §3.2). this model is based on greedy arc selection similar to the neural easy-first approach proposed in @xcite ) ( §3). we use tree-structured lstms to encode substructures (partial trees) in each formalism which are then concatenated across tasks and scored jointly. hence, we model interactions between substructures across tasks while keeping computational complexity low thanks to the easy-first framework. furthermore, this approach enables the sharing of various components between tasks, a richer sharing than the mere sequential encoder sharing found in most multitask systems @xcite . our multitask architecture outperforms the single-task parser on both formalisms ( §4.3). the parser itself is open-source and can be found at https://github.com/yash-reddy/mef_parser . beside efficiency, the tree-structured lstm easy-first framework provides several advantages which makes it appealing in our settings. new arc selection decisions are conditioned on encoded representations of partially parsed structures in both formalisms with the latest information at each step. since some word attachments are harder to find in one formalism than the other (longer range, ambiguous relations, etc.), we suppose that looking at the substructure involving such a word in one formalism may help make better decisions in the other. we do not need to postulate any priority between the tasks nor that all attachment decisions must be taken jointly which is computationally expensive, we leave the exact flow of information to be learned by the model. additionally, @xcite showed that even when no easy-first strategy is hard-wired into their multitask semantic dependency parser, it gets nevertheless learned from the data in a reinforcement learning framework. some possible enhancements to our model are explored in §6. ## model we briefly describe the single task easy-first (ef) parsing algorithm @xcite and its components, then we discuss the multitask extension (mef) inspired by @xcite adapted to the tree-structured lstm in terms of the updated parsing algorithm and its components. ## conclusion and future work we presented a dependency parsing model based on a multidimensional neural easy-first model @xcite . this architecture enables sharing representation at various levels of abstraction, and at different time steps of the parsing process, which makes it possible to communicate information and to learn when sharing information is important across dimensions. we tested this model on two syntactic dependency formalisms for arabic that vary on the morpho-syntactic (catib) to semantic (ud) spectrum. our experiments showed that this architecture gives a 9.9% error reduction on catib, and 6.1% error reduction on ud. further analysis of this reduction shows that its main contributor is not the sharing of lexical information, as is commonly done in multitask systems, but the sharing of partial dependency trees given as input for arc weight prediction. future work will explore further sharing between parsers. in particular, we expect tree encoders could benefit from the additional information (although redundancy with tree sharing in the score function could hurt the system). we also plan to work on model improvements to address the limitation arising from added dimensions. for instance, with the addition of the second dimension, the notion of oracle @xcite used for training becomes more fragile, since the number of correct actions grows, and the order in which to perform them is unknown and can have some long term consequences on the action in the other dimension. it would be interesting to explore how reinforcement learning, where the notion of planning ahead is crucial, could help.