Part-of-speech Tagging for Extremely Low-resource I ndian Languages (2024)

Abstract: Modern natural language processing (NLP) systems thrive when given access to large datasets. However, a large fraction of the world’s languages are not privy to such benefits due to sparse documentation and inadequate digital representation. This is especially true for Indian regional languages. As a first step towards expanding the reach of NLP technologies to extremely low-resource Indian languages, we present a new parallel part-of-speech (POS) evaluation dataset for Angika, Magahi, Bhojpuri and Hindi. Angika, Magahi, Bhojpuri, along with the more well-known Hindi, are all languages spoken in the Indian states of Bihar, Jharkhand and West Bengal. Ours is notably the first NLP resource, even for a shallow NLP task like POS-tagging, for Angika. We establish POS-tagging baselines using state-of-the-art multilingual pretrained language models (PLMs) finetuned on Hindi data, and show zero-shot evaluations on the other three languages. While all four languages use the same Devanagari script, pretrained tokenizers underperform in zero-shot on the three languages. We propose a simple look-back fix to address the tokenization challenge yielding F1-score improvements of up to 8% on Angika and show how it comes very close to an oracle setting when the underlying Hindi word is known (and can be accurately tokenized).

## introduction india is a multilingual country with more than 1369 languages and five main language families (office of the registrar general census commissioner, 2022). while the indian constitution officially recognizes 22 languages, numerous others face a battle for survival. english is spoken by only roughly 10% of the population (office of the registrar general census commissioner, 2022) in india; the majority prefers their diverse regional languages deeply rooted in cultural heritage. building technologies for regional indian languages is important to ensure inclusivity across user groups and empower people for everyday interactions. while there has been progress towards promoting multilinguality and linguistic diversity across indian languages with tools like indicnlpsuite @xcite and multilingual corpora such as common crawl oscar corpus @xcite , pmindia @xcite , and samanantar @xcite , there is almost no representation of low-resource indian languages like angika in these resources. in this work, we focus on three very lowresource indian languages angika, magahi and bhojpuri and create parallel pos-tagging evaluation corpora for these three languages, consistent with universal dependencies (ud) guidelines. these are the first ud-compliant datasets for angika and magahi. we also create a hindi pos-tagging dataset that is parallel to the data in the three languages. hindi is the closest highresource indian language that is related to bhojpuri, angika and magahi. this allows us to carefully examine the cross-lingual performance gap compared to hindi, whether transfer from a related high-resource language like hindi is possible, and challenges related to tokenization that affect zeroshot performance on the low-resource languages. we propose a simple look-back scheme that circumvents most errors that stem due to suboptimal tokenization for the three low-resource languages. to encourage further work on these languages, we publicly release our new dataset and code to reproduce all our experiments foot_0 . ## related work shallow nlp tasks such as pos tagging for lowresource languages have been studied fairly extensively in prior work, many of which have explored cross-lingual transfer learning techniques. fang and cohn (2016) achieve successful pos tagging, especially in low-resource languages (like malagasy and kinyarwanda), by combining word alignment with gold-standard data. @xcite implemented a bilstm model that utilizes word and character embeddings to transfer knowledge without relying on parallel corpora. another approach by @xcite introduces zero-shot tagging, by projecting annotations from a related highresource language, such as russian for ukrainian. more recently, dione et al. (2023) demonstrated significant improvements in pos tagging by using multilingual pretrained lms trained on typologically diverse african languages. recent work has also employed modular learning @xcite @xcite techniques with multilingual pretrained language models (plms) to enable effective crosslingual transfer to low-resource languages. prior work on pos tagging for various low-resource indo-aryan languages has mainly utilized classical nlp techniques and do not leverage plms. for example, @xcite developed a pos tagger for assamese and @xcite for awadhi, and both works used an hmm model. among the four languages of interest, other than hindi, there is an existing ud-compliant postagging evaluation dataset for bhojpuri (ojha and zeman, 2020). however, it only has 268 sentences. overall, there is a strong need for nlp datasets covering extremely low-resource indian languages (such as angika). we think it is also useful to develop parallel datasets along with relatively higherresource languages (such as hindi) that share common geographical boundaries, word order, script, and language family. this helps understand how cross-lingual transfer can be more effectively utilized from high-resource languages. ## data collection and annotation we created a parallel corpus comprising 708 evaluation sentences each for hindi, angika, bhojpuri, and magahi. we used some hindi monolingual data from @xcite and translated into angika, magahi, and bhojpuri. some angika monolingual data was extracted from the webonary foot_1 angika dictionary and translated into hindi and the other two languages. the ud dataset primarily draws data from the news domain, resulting in a higher frequency of named entities belonging to the noun class. to improve diversity, we incorporated sentences of daily conversations and regional stories. additionally, we included a few frequently used sentences by native speakers. issues in the webonary angika dictionary such as word misspellings and missing sentences were manually addressed. languages were translated by respective language annotators and verified by native speakers. ## experiments and results we provide pos tagging baselines using multilingual plms in a zero-shot setting. we finetune an indic-specific multilingual plm (muril @xcite ) on hindi ud pos-tagging data consisting of 13k sentences. we refer to this model as "hindi ft". we also fine-tuned on other massively multilingual plms, xlm-r-large @xcite and @xcite ), as a comparison to muril. all three plms have been pretrained on around 100-110 languages, with the exception of muril, which is exclusively pretrained on 12 major indic languages. however, none of these models are pretrained on any of the low-resource languages that we explore in this work. more details regarding the experimental setup can be found in appendix a.5. ## conclusion this paper introduces a ud-compliant parallel pos tag dataset for three extremely low-resource indian languages: angika, magahi and bhojpuri. this work contributes one of the first nlp resources for these languages and is a first step towards addressing their under-representation in the digital landscape. we provide state-of-the-art pos baselines by fine-tuning multilingual plms with hindi data. we find that pretrained indic tokenizers adversely affect cross-lingual transfer to angika, magahi and bhojpuri which we largely address with a simple look-back scheme.