Multilingual End-to-end Dependency Parsing with Linguistic Typology knowledge (2023)

Abstract: We evaluate a Multilingual End-to-end BERT based Dependency Parser which parses an input sentence by directly predicting the relative head-position for each word within it. Our model is a Cross-lingual dependency parser which is trained on a diverse polyglot corpus of high-resource source languages, and is applied on a low-resource target language. To make model more robust to typological variations between source and target languages, and to facilitate the cross-lingual transferring, we utilized the Linguistic typology knowledge, available in typological databases WALS and URIEL. We induce such typology knowledge within our model through an auxiliary task within Multi-task Learning framework.

## introduction linguistic typology is the classification of human languages according to their syntactic, phonological and semantic features. there are numerous available typological databases such as wals @xcite , sswl @xcite , lapsyd @xcite , autotyp @xcite , apcls (michaelis and magnus huber, 2013) etc. these databases provide taxonomies of typological features and their possible values, as well as the respective feature values for most of the world's languages. linguistic typology existed as an independent research domain since long @xcite @xcite but recently it has been used along with cross-lingual/multi-lingual nlp @xcite @xcite @xcite to address the issue of data-sparsity in low-resource languages. however all the popular typological databases suffer from a major shortcoming of limited coverage. in fact, values of many important typological features for most languages (specially less documented ones) are missing in these databases. this sparked a line of research on automatic acquisition of such missing typology knowledge. many researchers @xcite @xcite @xcite indeed successfully used multi-lingual nlp and ml techniques to predict these missing feature values. thus multilingual nlp and language typology feature prediction are very closely related tasks which would complement each other. based on this intuition, we propose a model that performs both multilingual nlp and linguistic typology feature prediction tasks simultaneously, in a multi-tasking setup. multi-task learning (mtl) @xcite is neural network framework which involves performing of two or more tasks simultaneously leading to knowledge/parameter sharing. these tasks are closely related thus complement each other leading to improved performance on all of them. even in scenarios where we primarily care about a single task, using a closely related task as an auxiliary task for mtl can be useful @xcite @xcite @xcite . in this work, we use linguistic typology feature prediction task as auxiliary task for end-to-end cross-lingual dependency parsing. hence, we make following contributions. ## related work cross-lingual model-transfer approaches to dependency parsing such as @xcite @xcite @xcite @xcite @xcite involve training a model on high-resource languages and subsequently adapting it to low-resource languages. participants of conll 2017 shared-task @xcite and conll 2018 shared task @xcite also provide numerous approaches to dependency parsing of low-resource languages. some approaches such as @xcite @xcite @xcite @xcite used typological information to facilitate crosslingual transfer. all these approaches directly feed the linguistic typology features into the model whereas we induce the linguistic typology knowledge through multitask learning. inducing typology knowledge through mtl rather than directly feeding it along with wordembeddings have following advantages. ## base end-to-end bert parser this section elaborates the details of our end2end bert based dependency parser which directly predicts the relative head position tag of each word within input sentence. given a sentence of length t, its dependency parsetree can be represented as a sequence of t relative head-position tags as demonstrated in figure 1a . figure 2a depicts the architecture of our baseline model. the depicted architecture comprises of three components namely bert encoder, output network and tree-decoder described as section 3.1, 3.2 and 3.3.