Improving the Performance of UD ify with Linguistic Typology Knowledge (2021)

Abstract: UDify is the state-of-the-art language-agnostic dependency parser which is trained on a polyglot corpus of 75 languages. This multilingual modeling enables the model to generalize over unknown/lesser-known languages, thus leading to improved performance on low-resource languages. In this work we used linguistic typology knowledge available in URIEL database, to improve the cross-lingual transferring ability of UDify even further.

## introduction state-of-the-art approaches to dependency parsing are supervised approaches that require large manually annotated dataset to be trained on, thus limiting their utility to only few high-resource languages. multilingual modeling which involves training a model on a mixed polyglot corpus of high-resource source-languages and applying it on a low resource target-language, is an effective way to circumvent this issue of data-sparsity. in a similar way, as the proficiency of a speaker's previous languages can enhance his/her ability to learn a new language @xcite , a model which is trained on multilingual dataset can learn to generalize over unknown or lesser-known languages. udify @xcite is the stateof-the-art mbert based language-agnostic dependency parser, which takes the advantage of multilingual modeling to improve its performance on low-resource languages. the authors of udify @xcite trained it on a joint polyglot corpus created by concatenating all training treebanks available in udv2.3, and evaluated it on all test treebanks in udv2.3 individually. results outlined in @xcite show that for dependency parsing task, the udify outperforms its baseline monolingual udpipe future @xcite model by a large margin especially for low-resource languages, as the model benefit significantly from the cross-lingual transfer learning which occurs due to joint polyglot training. however, the performance of udify model on the low-resource languages (less represented in the polyglot training corpus) is still much lower than the performance of it on the high-resource languages which are well represented within the training corpus. in this work, we use linguistic typology knowledge to improve the cross-lingual transferring ability of udify model even further, thereby significantly reducing this gap between model's performance on high-resource and low-resource languages. we induce the linguistic typology knowledge available in uriel @xcite database into the udify model by adding an auxiliary task of linguistic typology feature prediction to it, within the multitasking framework. sections 3 and 4 will describe the model in more details. ## related work cross-lingual model-transfer approaches to dependency parsing such as @xcite @xcite @xcite @xcite @xcite involve training a model on high-resource languages and subsequently adapting it to low-resource languages. participants of conll 2017 shared-task @xcite and conll 2018 shared task @xcite also provide numerous approaches to dependency parsing of low-resource languages. some approaches such as @xcite @xcite @xcite @xcite indeed used linguistic typology to facilitate the cross-lingual transfer between source and target languages. however, all these approaches directly feed the linguistic typology features into the respective model, whereas we induce the linguistic typology knowledge into udify model through multitask learning. inducing typology knowledge through multitask learning rather than directly feeding it along with word-embeddings have following advantages. ## udify udify is a multitasking multilingual bert based model which performs four key languageprocessing tasks simultaneously namely upostagging, ufeat-tagging, lemmatization and dependency parsing, in a multitasking framework. the model utilizes a single shared mbert based encoder, and four individual task-specific decoders, for each of the four tasks respectively. the mbert encoder takes in the entire sentence as input, tokenizes it using pre-trained wordpiece tokenizer @xcite and subsequently outputs mbert (wu and dredze, 2019) based contextualized-embeddings for each word within the input-sentence. we refer to original udify (kondratyuk and straka, 2019) paper for detailed description of mechanism of computing/fine-tuning such contextualized embeddings. the decoders for both upos-tagging and ufeattagging tasks adopt standard sequence-tagging architecture with softmax layer on the top. these decoders accept the contextual embeddings generated from the mbert encoder for each word in the input sentence, and predicts its upos/ufeats tag. for lemmatization task as well, the model uses a standard sequence-tagger which predicts a classtag representing a unique edit script, for each word. an edit-script is simply the sequence of character operations to transform a word form to its lemmaform. for dependency-parsing, the model adopts the pop-ular deep biaffine architecture @xcite for graph-based parsing, with lstmencoder been replaced by the shared mbert encoder. ## linguistic typology prediction to improve the cross-lingual transferring ability of udify model, we added a fifth auxiliary task of linguistic typology prediction to it. our typology-predictor is a simple deep feedforward neural network with sigmoid activation function, which predicts the values of all typology features provided by the uriel database @xcite . uriel database is a collection of binary features extracted from multiple typological, phylogenetic, and geographical databases such as wals @xcite , phoible (moran and richard wright, 2014), ethnologue (m. paul lewis and fennig, 2015) and glottolog @xcite . uriel database can be accessed through pyton pypi library called lang2vec foot_0 . let n be the number of typology features provided by uriel database. our typology predictor would then output the probability vector p r @xmath0. here @xmath1. @xmath2. p r t y comprises of the probability of value of each uriel binary feature being as 1, for the specific language being parsed. the total-loss is computed by simply adding the typology predictor loss to udify model's (as computed in (kondratyuk and straka, 2019)) ## experiments this section describes the details of experiments conducted to evaluate our proposed model. ## discussion it is observed that the udify-w-syntax variant of our proposed model outperforms the other two variants of it, for most of the test-treebanks, despite the fact that the udify-w-syntax+semantic and udifyw-all variants utilizes more typology-features than the udify-w-syntax variant. the reason being that since all four tasks performed by the udify model namely upos-tagging, ufeats-tagging, lammelization and dependency parsing are syntactic tasks, only the syntactic typology-features are relevant to these tasks. @xcite proved that, having large number of unrelated features makes it difficult for a neural-network model to effectively learn from provided training-data, and thereby would lead to drop in performance. ## conclusion in this work we used linguistic typology knowledge available in uriel database to improve the cross-lingual transferring ability of the state-of-theart language-agnostic udify parser. we injected typology knowledge in udify model through an auxiliary task, in multitasking settings. long duong, trevor cohn, steven bird, and paul cook. 2015. low resource dependency parsing: crosslingual parameter sharing in a neural network parser. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 2: short papers), pages 845-850. agnieszka falenska and özlem çetinoglu. 2017. lexicalized vs. delexicalized parsing in low-resource scenarios. in proceedings of the 15th international conference on parsing technologies, pages 18-24. jiang guo, wanxiang che, david yarowsky, haifeng wang, and ting liu. 2016. a representation learning framework for multi-source transfer parsing. in thirtieth aaai conference on artificial intelligence. martin haspelmath harald hammarstrom, robert forkel and sebastian bank. 2015. glottolog 2.6. martin haspelmath. 2009. the typological database of the world atlas of language structures. berlin: walter de gruyter. james henderson. 2004. discriminative training of a neural network statistical parser. in acl'04: proceedings of the 42nd annual meeting of the association for computational linguistics, pages 95-102. association for computational linguistics. dan kondratyuk and milan straka. 2019. 75 languages, 1 model: parsing universal dependencies universally. arxiv preprint arxiv:1904.02099. patrick littell, david r mortensen, ke lin, katherine kairis, carlisle turner, and lori levin. 2017. uriel and lang2vec: representing languages as typological, geographical, and phylogenetic vectors. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 2, short papers, pages 8-14. gary f. simons m. paul lewis and charles d. fennig. 2015. ethnologue: languages of the world, eighteenth edition. ryan mcdonald, slav petrov, and keith b hall. 2011. multi-source transfer of delexicalized dependency parsers. daniel mccloy moran, steven and editors richard wright. 2014. phobia online. phoebe mulcaire, jungo kasai, and noah a smith. 2019. low-resource parsing with crosslingual contextualized representations. arxiv preprint arxiv:1909.08744. tahira naseem, regina barzilay, and amir globerson. 2012. selective sharing for multilingual dependency parsing. the association for computational linguistics. mohammad sadegh rasooli and michael collins. 2017. cross-lingual syntactic transfer with limited resources. transactions of the association for computational linguistics, 5:279-293. ehsan shareghi, yingzhen li, yi zhu, roi reichart, and anna korhonen. 2019. bayesian learning for neural dependency parsing. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 3509-3519. milan straka. 2018. udpipe 2.0 prototype at conll 2018 ud shared task. in proceedings of the conll 2018 shared task: multilingual parsing from raw text to universal dependencies, pages 197-207. oscar täckström, ryan mcdonald, and joakim nivre. 2013. target language adaptation of discriminative transfer parsers. clara vania, yova kementchedjhieva, anders søgaard, and adam lopez. 2019. a systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages. arxiv preprint arxiv:1909.02857. david vilares, carlos gómez-rodríguez, and miguel a alonso. 2015. one model, two languages: training bilingual parsers with harmonized treebanks. arxiv preprint arxiv:1507.08449. dingquan wang and jason eisner. 2016a. the galactic dependencies treebanks: getting more data by synthesizing new languages. transactions of the association for computational linguistics, 4:491-505. dingquan wang and jason eisner. 2016b. the galactic dependencies treebanks: getting more data by synthesizing new languages. transactions of the association for computational linguistics, 4:491-505. shijie wu and mark dredze. 2019. beto, bentz, becas: the surprising cross-lingual effectiveness of bert. arxiv preprint arxiv:1904.09077. yonghui wu, mike schuster, zhifeng chen, quoc v le, mohammad norouzi, wolfgang macherey, maxim krikun, yuan cao, qin gao, klaus macherey, et al. 2016. google's neural machine translation system: bridging the gap between human and machine translation. arxiv preprint arxiv:1609.08144. daniel zeman, jan hajic, martin popel, martin potthast, milan straka, filip ginter, joakim nivre, and slav petrov. 2018. conll 2018 shared task: multilingual parsing from raw text to universal dependencies. in proceedings of the conll 2018 shared task: multilingual parsing from raw text to universal dependencies, pages 1-21.