Systematic Investigation of Strategies Tailored for Low-Resource Settings for Low-Resource Dependency Parsing (2023)

Abstract: In this work, we focus on low-resource dependency parsing for multiple languages. Several strategies are tailored to enhance performance in low-resource scenarios. While these are well-known to the community, it is not trivial to select the best-performing combination of these strategies for a low-resource language that we are interested in, and not much attention has been given to measuring the efficacy of these strategies. We experiment with 5 low-resource strategies for our ensembled approach on 7 Universal Dependency (UD) low-resource languages. Our exhaustive experimentation on these languages supports the effective improvements for languages not covered in pretrained models. We show a successful application of the ensembled system on a truly low-resource language Sanskrit. The code and data are available at: https://github.com/Jivnesh/SanDP

## introduction recently, the supervised learning paradigm has dramatically increased the state-of-the-art performance for the dependency parsing task for resourcerich languages @xcite @xcite @xcite . however, only a handful of resource-rich languages are able to take advantage, and many low-resource languages are far from these benefits @xcite @xcite . in literature, several strategies have been proposed to enhance performance in low-resource scenarios, such as data augmentation @xcite , cross/monolingual pretraining @xcite @xcite , sequential transfer learning @xcite , multitask learning @xcite , crosslingual transfer @xcite and selftraining @xcite . however, not much attention has been given to measuring the efficacy of the existing lowresource strategies well-known to the community for low-resource dependency parsing @xcite . this is essential to assess their utility for low-resource languages @xcite before inventing novel ways to tackle data sparsity. in this work, we systematically explore 5 pragmatic strategies for low-resource settings on 7 languages. we experiment with low-resource strategies such as data augmentation, sequential transfer learning, cross/mono-lingual pretraining, multitask learning and self-training. we investigate: (1) how is the trend in performance of each strategy across various languages? whether the choice of best performing variant of each strategy is language dependent? (2) we integrate the best performing variant of each strategy and call the resulting system as the ensembled system. do all the strategies contribute towards performance gain in the ensembled system? how well does this ensemble approach generalize across multiple low-resource languages? (3) how far can we push a purely datadriven ensemble system using the best-performing low-resource strategies? can this simple ensemble approach outperform the state-of-the-art of a low-resource language? we argue that while it may sound like a simple application of techniques well known to the community; it is non-trivial to select the best performing combination for a target low-resource language. our exhaustive experimentation empirically establishes the effective generalization ability of the ensembled system on 7 languages and shows average absolute gains of 5.2/6.2 points unlabelled/labelled attachment score (uas/las) over strong baseline @xcite . notably, our ensembled system shows substantial improvements for the languages not covered in pretrained models. finally, we show a successful application of the ensembled system on a truly low-resource language sanskrit. we find that the ensembled system outperforms the state-of-the-art system @xcite for sanskrit by 1.2 points absolute gain in terms of uas and shows comparable performance in terms of las ( § 3). ## investigation of strategies tailored for low-resource settings we explore 5 strategies specially tailored for lowresource settings on 7 languages, and integrate the best performing strategy of each category in our ensembled system ( language selection criteria: we choose lowresource languages with less than 2,500 training samples from 4 different typological families such that each language belongs to a unique sub-family. in order to accommodate a low-resource tailored pretraining @xcite , we choose languages that have explicit morphological information. additionally, we divide the set of languages into the languages covered/not-covered in the multilingual language model's pretraining: (1) covered: arabic (ar), greek (el), hungarian (hu) (2) not covered: wolof (wo), gothic (got), coptic (cop) and sanskrit (san). figure 1 illustrates the number of training samples available for all ud languages in ud-2.6. hedderich et al. (2021) ask "how low is low-resource?" and suggest that the threshold of low-resource is task and language-dependent. the low-resource settings can be seen as a continuum of resource availability due to the absence of a hard threshold. more efforts should focus on evaluating lowresource strategies across multiple languages for a fair comparison between these strategies. therefore, we select a threshold for the languages with less than 2,500 training samples (figure 1 ). we restrict ourselves to the setting where the target lowresource language does not have a high-resource related language that could possibly facilitate the positive cross-lingual or zero-shot transfer @xcite @xcite @xcite . thus, we do not consider low-resource languages with only a test set available. also, we do not consider cross-lingual transfer @xcite @xcite strategy in our study. dataset and metric: for each of these 7 lowresource languages from universal dependencies (ud-2.6) (de marneffe et al., 2021), following rotman and reichart (2019), we use 500 data points for training (allocating equal power to each language for fair comparison) and the original dev/test split as dev/test set. additionally, 1000 morphologically tagged data points (without dependency annotations) are used for self-training and pretraining. we use sentence level macro averaged uas/las metric for evaluation. hyper-parameters: for seqtral variants, we use the exact same encoder as @xcite with 2 bi-lstm layers and decoder with fully connected layer followed by softmax layer. for the ensembled system, we adopt biaff's codebase by @xcite with the hyper-parameters setting as follows: the batch size as 16, training iterations as 100, a dropout rate as 0.33, the number of stacked bi-lstm layers as 2, learning rate as 0.002 and the remaining parameters as the same as @xcite . we release our codebase publicly under creative-common licence. computing infrastructure used: we primarily use rtx-2080, 12 gb gpu memory, 4352 gpu cores computing infrastructure for our experiments. the second last block shows ablations when the best variant from each family is added to the ensembled system. for example, +data. aug. refers to the system with the best variant from all 5 strategies. the best performing system as per dev set is finally compared with biaff on the test set. xlm-r is not compatible with 3 languages and case information of gothic (got) language is missing; hence we do not report their results. cohen, 1989). seqtral-fe: we treat newly integrated layers as feature extractors (fe) by freezing them. seqtral-uf: gradually unfreeze (uf) these new layers in the top to down order @xcite . seqtral-dl: the discriminative learning rate (dl) is used for newly added layers @xcite , the learning rate is decreased from top-to-bottom layers. seqtral-ft: the default learning rate is used to fine-tune all newly added layers. ## application on sanskrit data: we use two standard benchmark datasets available for sanskrit. we use 1,700, 1,000 and 1,300 sentences (prose domain) from the sanskrit treebank corpus @xcite as train, dev and test set, respectively. we also evaluate on the vedic sanskrit treebank @xcite consisting of 1,500 , 1,024 and 1,473 sentences (poetry-prose mixed) as train, dev and test data, respectively. for both data, the final results on the test set are reported using systems trained with combined gold train and dev set. the ensembled system: figure 2 shows the ensembled system for sanskrit as per ## conclusion and discussion we focused on low-resource dependency parsing for multiple languages. we found that our ensembled system can benefit the languages not covered in pretrained models. while multi-lingual pretraining (mbert and xlm-r) is helpful for the languages covered in pretrained models, lcm pretraining (which simply uses an additional 1,000 morphologically tagged data points) is helpful for the remaining languages. thus, these findings would help community to pick strategies suitable for their language of interest and come up with robust parsing solutions. specifically for sanskrit, our ensembled system superseded the performance of the state-of-the-art hybrid system mg-ebm* by 1.2 points (uas) absolute gain and showed comparable performance in terms of las. limitations: we could not evaluate on complete ud due to limited available compute resources (single gpu), hence we selected 7 representative languages for our experiments.