A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing Task for Low-resource Morphologically Rich Languages (2021)

Abstract: Neural dependency parsing has achieved remarkable performance for many domains and languages. The bottleneck of massive labelled data limits the effectiveness of these approaches for low resource languages. In this work, we focus on dependency parsing for morphological rich languages (MRLs) in a low-resource setting. Although morphological information is essential for the dependency parsing task, the morphological disambiguation and lack of powerful analyzers pose challenges to get this information for MRLs. To address these challenges, we propose simple auxiliary tasks for pretraining. We perform experiments on 10 MRLs in low-resource settings to measure the efficacy of our proposed pretraining method and observe an average absolute gain of 2 points (UAS) and 3.6 points (LAS).

## introduction dependency parsing has greatly benefited from neural network-based approaches. while these approaches simplify the parsing architecture and eliminate the need for hand-crafted feature engineering @xcite @xcite @xcite , their performance has been less exciting for several morphologically rich languages (mrls) and low-resource languages @xcite . in fact, the need for large labeled treebanks for such systems has adversely affected the development of parsing solutions for low-resource languages @xcite . @xcite observe that data-driven parsing on 9 low resource treebanks resulted not only in low scores but those outputs "are hardly useful for downstream applications". several approaches have been suggested for improving the parsing performance of low-resource languages. this includes data augmentation strategies, cross-lingual transfer @xcite and using unlabelled data with semi-supervised learning @xcite and self-training @xcite . further, incorporating morphological knowledge substantially improves the parsing performance for mrls, including lowresource languages @xcite . this aligns well with the linguistic intuition of the role of morphological markers, especially that of case markers, in deciding the syntactic roles for the words involved @xcite @xcite . however, obtaining the morphological tags for input sentences during run time is a challenge in itself for mrls @xcite and use of predicted tags from taggers, if available, often hampers the performance of these parsers. in this work, we primarily focus on one such morphologicallyrich low-resource language, sanskrit. we propose a simple pretraining approach, where we incorporate encoders from simple auxiliary tasks by means of a gating mechanism @xcite . this approach outperforms multitask training and transfer learning methods under the same low-resource data conditions (∼500 sentences). the proposed approach when applied to @xcite , a neural parser, not only obviates the need for providing morphological tags as input at runtime, but also outperforms its original configuration that uses gold morphological tags as input. further, our method performs close to dcst @xcite , a self-training based extension of @xcite , which uses gold morphological tags as input for training. to measure the efficacy of the proposed method, we further perform a series of experiments on 10 mrls in low-resource settings and show 2 points e i fc + dropout softmax devāḥ asurāḥ vai eṣu lokeṣu samayatanta h devāḥ h asurāḥ h vai h eṣu h lokeṣu h samayatanta nom.pl.m. nom nsubj nom.pl.m. nom conj adv adv discourse loc.pl.m. loc det loc.pl.m. loc obl past.pl.3 verb root (a) h p h 1 h 2 h foot_1 h p h 1 devāḥ h 2 h 3 e p e 1 e 2 e 3 e p e 1 e 2 e 3 asurāḥ mlp mlp' (3) are the encoders pre-trained with proposed auxiliary tasks. gating mechanism combines representations of all the encoders which, for each word pair, is passed to two mlps to predict the probability of arc score (s) and label (l). and 3.6 points average absolute gain ( § 3.1) in terms of uas and las, respectively. our proposed method also outperforms multilingual bert (devlin et al., 2019, mbert) based multi-task learning model (kondratyuk and straka, 2019, udify) for the languages which are not covered in mbert ( § 3.4). ## pretraining approach our proposed pretraining approach essentially attempts to combine word representations from encoders trained on multiple sequence level supervised tasks, as auxiliary tasks, with that of the default encoder of the neural dependency parser. while our approach is generic and can be used with any neural parser, we use biaffine parser @xcite , hence forth referred to as bi-aff, in our experiments.this is a graph-based neural parser that makes use of biaffine attention and a biaffine classifier. 2 figure 1 illustrates the proposed approach using an example sequence from sanskrit. our pipeline-based approach consists of two steps: (1) pretraining step (2) integration step. figure 1a describes the pretraining step with three auxiliary tasks to pretrain the corresponding encoders e (1)-(3) . finally, in the integration step, these pretrained encoders along with the encoder for the biaff model e (p ) are then combined us-2 more details can be found in supplemental ( § a.1). for all the models, input representation consists of fasttext @xcite foot_2 embedding of 300-dimension and convolutional neural network (cnn) based 100-dimensional character embedding @xcite . for character level cnn architecture, we use following setting: 100 number of filters with kernel size equal to 3. we use standard unlabelled and labelled attachment scores (uas, las) to measure the parsing performance and use t-test for statistical significance @xcite . for stbc treebank, the original data does not have morphological tag entry, so the sanskrit heritage reader @xcite is used to obtain all the possible morphological analysis and only those sentences are chosen which do not have any word showing homonymy or syncretism @xcite . for other mrls, we restrict to the same training setup as sanskrit and use 500 annotated sentences as labeled data for training. additionally, we use 1000 sentences with morphological information as unlabelled data for pretraining sequence taggers. 5 we use all the sentences present in original development and test split data for development and test data. for languages where multiple treebanks are available, we chose only one available treebank to avoid domain shift. note that stbc adopts a tagging scheme based on the grammatical tradition of sanskrit, specifically based on kāraka @xcite , while the other mrls including sanskrit-vedic use ud. hyper-parameters: we utilize the biaffine parser (biaff) implemented by @xcite . we employ the following hyper-parameter setting for pretraining sequence taggers and base parser biaff: the batch size of 16, number of epochs as 100, and a dropout rate of 0.33 with a learning rate equal to 0.002. the hidden representation generated from n-stacked-lstm layers of size 1,024 is passed through two fully connected layers of size 128 and 64. note that lcm and mtl models use 2-stacked lstms. we keep all the remaining parameters the same as that of @xcite . for all transeq variants, one bilstm layer is added on top of three augmented pretrained layers from an off-the-shelf morphological tagger @xcite to learn task-specific features. in transeq-fea, the dimension of the non-linearity layer of the adaptor module is 256, and in transeq-uf, after every 20 epochs, one layer is unfrozen from top to down fashion. in transeq-dl, the learning rate is decreased from top to down by a factor of 1.2. we have used default parameters to train hierarchical tagger foot_4 and baseline models. models: all our experiments are performed as augmentations on two off the shelf neural parsers, biaff @xcite and deep contextualized self-training (dcst), which integrates self-training with biaff (rotman and reichart, 2019). 7 hence their default configurations become the baseline models (base). we also use a system that simultaneously trains the biaff (and dcst) model for dependency parsing along with the sequence level case prediction task in a multi task setting (mtl). for mtl model, we also experiment with morphological tagging, as an auxiliary task. however, we do not find significant improvement in performance compared to case tagging. hence, we consider case tagging as an auxiliary task to avoid sparsity issue due to the monolithic tag scheme for morphological tagging. as a transfer learning variant (transeq), we extract first three layers from a hierarchical multi-task morphological tagger @xcite , trained on 50k examples from dcs @xcite . here each layer corresponds to different grammatical categories, namely, number, gender and case. note that number of randomly initialised encoder layers in biaff (and dcst) are now reduced from 3 to 1. we fine-tune these layers with default learning rate and experiment with four different fine-tuning schedules. 8 finally, our proposed configuration (in §2) is referred to as the lcm model. 9 we also train a version each of the base models which expects morphological tags as input and is trained with gold morphological tags. during runtime, we report two different settings, one which uses predicted tags as input (predicted mi) and other that uses gold tag as input (oracle mi). we obtain the morphological tags from a neural crf tagger @xcite trained on our training data. oracle mi will act as an upper-bound on the reported results. ## conclusion in this work, we focused on dependency parsing for low-resource mrls, where getting morphological information itself is a challenge. to address low-resource nature and lack of morphological information, we proposed a simple pretraining method based on sequence labeling that does not require complex architectures or massive labelled or unlabelled data. we show that little supervised pretraining goes a long way compared to transfer learning, multi-task learning, and mbert pretraining approaches (for the languages not covered in mbert). one primary benefit of our approach is that it does not rely on morphological information at run time; instead this information is leveraged using the pretrained encoders. our experiments across 10 mrls showed that proposed pretraining provides a significant boost with an average 2 points (uas) and 3.6 points (las) absolute gain compared to dcst. in the transeq setting, as we move down across its sub-variants in