Cross-lingual Strategies for Low-resource Language Modeling: A Study on Five I ndic Dialects (2023)

Abstract: Neural language models play an increasingly central role for language processing, given their success for a range of NLP tasks. In this study, we compare some canonical strategies in language modeling for low-resource scenarios, evaluating all models by their (finetuned) performance on a POS-tagging downstream task. We work with five (extremely) low-resource dialects from the Indic dialect continuum (Braj, Awadhi, Bhojpuri, Magahi, Maithili), which are closely related to each other and the standard mid-resource dialect, Hindi. The strategies we evaluate broadly include from-scratch pretraining, and cross-lingual transfer between the dialects as well as from different kinds of off-the- shelf multilingual models; we find that a model pretrained on other mid-resource Indic dialects and languages, with extended pretraining on target dialect data, consistently outperforms other models. We interpret our results in terms of dataset sizes, phylogenetic relationships, and corpus statistics, as well as particularities of this linguistic system.

## introduction in the last decade, natural language models have made tremendous progress on multiple tasks @xcite . many recent advances in natural language processing (nlp) owe credit to neural models that are pretrained over large quantities of unlabeled text, such as bert @xcite . data inequity over the vast range of the world's languages has led efforts to "transfer" these data-hungry neural models from resource-rich languages such as english and spanish, to lower-resource languages @xcite , with many studies focusing on phylogenetic closeness between the source and target languages as one of the important factors determining the results @xcite @xcite . indic nlp, or nlp for indian languages, 1 has also made corresponding advances, with the release of large corpora, language models, and benchmarks for 18 "major" indian languages @xcite . however, there are hundreds of other languages and dialects in india, many of them spoken by millions of people, such as rajasthani, kannauji, garhwali, and others, that have non-existent or nascent nlp research @xcite . we work with a typical real-world situation, with five (extremely) low-resource north indian dialects belonging to the indic language family-namely, braj, awadhi, bhojpuri, magahi, maithili. these languages bear close relationships with hindi, a mid-resource dialect 2 although with a number of morphosyntactic and lexical divergences. we compare strategies for language modeling for lowresource languages, using a part-of-speech (pos) tagging downstream task for evaluation. we report that relatedness at the level of the language family between the pretraining languages and the target language benefits downstream performance. however, while comparing low-resource dialects as sources for a particular target dialect in a transfer setup, the results are less explained by phylogeny than by corpus domain match and lexical overlap. finally, we find that extended pretraining shows consistent benefits. we hope that these experiments will raise an interest in nlp for these dialects, and constitute a starting point for other work in this context. ## related work recently, there have been some attempts to develop basic nlp tools and resources for some of the prominent languages of the indic continuum. mundotiya et al. @xcite collect monolingual data and pos-tagged corpora for bhojpuri, maithili, and magahi, also providing crf baselines for pos tagging. @xcite collect a monolingual corpus for maithili as well as some pos-annotated data, 3 ojha (2019) contribute a similar effort for bhojpuri, and ojha et al. (2020) 1 the word "indic", depending on context, is used to refer to both the subfamily of the indo-european family spoken in india (e.g. hindi, bengali, marathi), and the indian languages in general (including non-indo-european languages such as the dravidian family of languages). in this paper, unless otherwise mentioned, we use the first, phylogenetic, sense of the term. ## experiments we compare different multilingual or cross-lingual transfer strategies in the context of indic dialects, evaluated on the downstream task of pos-tagging. we will refer to the two following operations: "pretraining" refers to training a model on mono/multilingual raw text, "ep" or "extended pretraining" refers to further pretraining an already pretrained model on different mono/multilingual raw text. 10 all models are finetuned and evaluated on the annotated dataset of the target dialect. due to tagset differences, we cannot attempt zero-shot transfer without performing label alignment. finally, we do not pretrain from scratch on hindi data and use a publicly available hindi pretrained model instead @xcite . we use the huggingface transformers library @xcite for training and accessing publicly available pretrained models. all models (including the publicly available models that we use) have a bert-base architecture, consisting of 12 attention heads, 12 layers, and with hidden layer size 768. models trained from scratch on dialect data are trained on the masked language modeling (mlm) objective for ~40 epochs, ep over pretrained models is performed for 15 epochs, and finetuning is performed over the best performing pretraining checkpoint for 5 epochs. 11 baseline as the baseline for each dialect, we use the bert architecture described above, pretrained from scratch on monolingual dialect data, and finetuned on task data for the dialect. using pretrained models we report the results obtained by finetuning three publicly available large pretrained models on the pos-tagged dataset for each dialect: we also perform extended pretraining for the muril and hin-bert models to observe potential benefits. specifically, these models are pretrained further with an mlm objective on the monolingual data of the target dialect, and then (like all other models) finetuned and evaluated on the target dialect. the muril model was chosen over mbert due to its better initial performance. 15 using related dialects one can use data in related low-resource dialects to boost the learning of shared properties and similar words. we conduct the following experiments to investigate this idea: ## results and discussion monolingual performance within dialects, differences in performance can be explained by the size of the annotated dataset, the amount of monolingual data, and the complexity of the tagset (see bhojpuri and magahi, the highest resourced dialects, have baseline performances roughly on par with {mbert | muril}+ft. although these dialects are still very low-resource by several orders of magnitude compared to hindi, their datasets are already big "enough" to yield a good performance on a relatively shallow task such pos-tagging. the transfer methods are mainly beneficial for the lowest-resourced dialects, braj, awadhi, and maithili. pretrained models comparing {hin-bert | muril | mbert}+ft, we observe that muril-based models do better than mbert-based models on four out of six dialects. this extends the results shown by @xcite that demonstrate that muril outperforms mbert consistently on its 17 pretraining languages. we also see that hin-bert and mbert seem to perform on par, with perhaps a slight edge to hin-bert, although mbert is pretrained on much more data. this corroborates the intuition that the relatedness of the pretraining languages with target languages could positively affect transfer results, "compensating", in a way, for less data. however, the fact that these pretrained models differ in their pretraining objectives must be kept in mind while making observations about the effects of pretraining languages. 17 the increase in performance over the monolingual baseline with pretrained models, especially for hin-bert+ft, can also be contextualized by crosslingual lexical similarity between monolingual corpora (see figure 2a ). we see that braj, awadhi, and bhojpuri show the highest lexical similarity with hindi. this explains the jump in performance for low-resource braj and awadhi (whereas for bhojpuri, which already has a good amount of monolingual data, training on hindi causes worsening). extended pretraining ep helps consistently; language-specific pretraining possibly serves to expose the model to non-cognate words or language specific constructions in the target language. the only performance drop is observed for maithili; monolingual ep slightly worsens performance in both hin-bert+ft and muril+ft. this accords with our earlier observation of the low maithili mono-pos overlap, and its possible effects. using other dialect data the best performing single-language transfer from another low-resource dialect, i.e. the d+ft model, does better than the monolingual model for dialects with little monolingual data, namely, braj, maithili and awadhi, and worse for the higher resourced dialects i.e. bhojpuri and magahi. abbmm+ft always does better than d+ft, presumably because the model sees monolingual data in the target dialect as well as more related dialect data in general. we also observe that abbmm+ft is on par with {mbert | muril}+ft even for dialects without much monolingual data; again, this indicates that models pretrained on roughly of a few million tokens, even from closely related dialects, perform comparably with much larger (language family or other) multilingual models (pretrained on three orders of magnitude more data) for a downstream pos-tagging task. it is possible that this is not the case for tasks requiring more language understanding. pairwise transfer with equal dataset size by fixing dataset sizes for all dialects, we aim to directly compare different dialects as (pretraining) sources for a given target dialect. 18 the resulting f1 scores are presented in we also use these scores ( takeaways the takeaways from the results can be summarized as follows: ## conclusion in this paper, we looked at different strategies for developing language models for low-resource languages, using five extremely low-resource dialects belonging to the indic continuum as a testbed. we compared conventional pretraining and cross-lingual transfer methods, and concluded that large pretrained models trained on the same language family (in our case muril, for the indic language family) are particularly successful as base models, especially if followed by extended pretraining, either monolingually or on closely related dialect data. we hope that this work contributes to building a basic research base for the indic dialect continuum, as well as other dialect systems.