Evaluating Cross Lingual Transfer for Morphological Analysis: a Case Study of I ndian Languages (2023)

Abstract: Recent advances in pretrained multilingual models such as Multilingual T5 (mT5) have facilitated cross-lingual transfer by learning shared representations across languages. Leveraging pretrained multilingual models for scaling morphology analyzers to low-resource languages is a unique opportunity that has been under-explored so far. We investigate this line of research in the context of Indian languages, focusing on two important morphological sub-tasks: root word extraction and tagging morphosyntactic descriptions (MSD), viz., gender, number, and person (GNP). We experiment with six Indian languages from two language families (Dravidian and Indo-Aryan) to train a multilingual morphology analyzers for the first time for Indian languages. We demonstrate the usability of multilingual models for few-shot cross-lingual transfer through an average 7% increase in GNP tagging in a cross-lingual setting as compared to a monolingual setting through controlled experiments. We provide an overview of the state of the datasets available related to our tasks and point-out a few modeling limitations due to datasets. Lastly, we analyze the cross-lingual transfer of morphological tags for verbs and nouns, which provides a proxy for the quality of representations of word markings learned by the model.

## introduction morphology analysis is the first step of processing in the classical nlp pipeline. even in the transformer era, wherein the entire nlp pipeline is replaced with a transformer, the use of morphological segmentation for tokenization instead of statistical subword tokenization has been shown to produce better embeddings, especially for morphologically rich languages @xcite . the statistical subword tokenization used in tokenizers such as wordpiece cannot capture * work done while at iit bombay morphological alternations (e.g. wordpiece doesn't treat contextual allomorphs as related) and nonconcatenative morphology @xcite . one of the tasks that we analyze in our work is root word extraction, which forms an integral component of morphologically informed segmentation. a morphology analyzer can also help speed up language documentation efforts for endangered languages, @xcite leveraged interlinear glossed text to generate unseen forms of inflectional paradigm using a morphology analyzer. availability of morphological information can also benefit various downstream tasks such as parsing @xcite , machine translation @xcite , language modeling @xcite , etc. our scope of this work is inflectional and concatenative morphology. we also envision our work to be used in bias-aware machine translation, especially from morphologically poor languages to morphologically richer languages. for example, if we want to translate the sentence "my friend was a doctor" to hindi, we would ideally prefer to have both masculine and feminine translations "mera dost doctor tha" (masculine) and "meri dost doctor thi"(feminine), as english sentence has no mention of gender and for hindi, the gender markers are present on verbs (tha\thi) and pronouns (mera\meri). although high-quality morphology analyzers have been built for some indian languages, they are either rule-based such as @xcite , or are neural models trained on annotated data which is available in sufficient quantities only for high resource languages @xcite . building morphology analyzers for low-resource languages remains a challenging task. for low-resource languages, morphological resources are sparse or virtually nonexistent. multilingual models have shown promising results for cross-lingual transfer from high-resource to low-resource languages @xcite percentage of data points with a particular feature marking is present gender number person tense aspect case modality others 60.4 94.8 82.1 58.5 35.5 11.0 11.0 27. 5 in this work, we study the multilingual capability of mt5 @xcite to carry out cross-lingual transfer of morphological features and extract the root words given the surface forms. we also test the multilinguality hypothesis that, in the presence of annotated examples of source languages, the required number of annotated examples of the target language to get identical results reduces. we carry out this analysis of cross-lingual transfer within language families and across (language) families and provide pointers to effective usage of multilingual data. the languages we carry out morphological analysis are of the dravidian family (tamil, telugu, and kannada) and the indo-aryan family (bengali, hindi, and marathi). we also give a brief account of the state of datasets available for morphological analysis and their challenges. we finetune mt5 for gender, number, and person tagging for verbs and nouns in six indian languages: marathi, hindi, bengali, tamil, telugu, and kannada. the features: gender, number, and person (gnp) are hereby referred to as morphosyntactic description (msd) tags. the current state of the datasets and inconsistency of annotation across languages limits our analysis to gnp tags of verbs and nouns. our contributions are as follows: morphological analysis: for morphological analysis, sigmorphon @xcite has been one of the venues organizing shared tasks and workshops related to computational morphology and multilingual morphological analysis, especially in the low resource scenarios. shared tasks such as @xcite @xcite , etc. looked at morphological reinflection with an increasing number of languages each year. for morphological reinflection, the output is the surface form, and the inputs are: a root word (or any other form of the root word) and desired features in the surface form (the output). task 2 in cotterell et al. (2018) as well as mccarthy et al. (2019) explored morphological analysis and reinfection in context. jin et al. (2020) and wiemerslage et al. (2021) were aimed at unsupervised clustering of paradigms, wherein given a lemma list, the goal is to output all the possible forms of a lemma. morphosyntactic lexicon generation is one task closely related to morphological analysis; faruqui et al. (2016) used graph-based semi-supervised learning for label propagation. hulden et al. (2014) used a semi-supervised approach for lexicon construction from concrete inflection tables by generalizing the inflection paradigms from the tables provided. for morphology resources, apart from unimorph @xcite , the morphynet database @xcite is a large dataset of methodologically annotated surface forms spanning 15 languages and is extracted from wiktionary. there have also been efforts to create task-specific models for various components of cross-lingual morphological tagging @xcite indian language morphology: regarding resources for indian languages, @xcite points out resource scatteredness (rather than scarcity) as the primary obstacle to developing south asian language technology and proposes the study of language history and contact as one of the potential solutions. workshops like dravidian-langtech @xcite and wildre @xcite are dedicated specifically to the development of technologies and resources for indian languages. the unimorph database (mccarthy et al., 2020) has been one of the recent efforts to extend the coverage of computational morphological resources. cotterell and heigold (2017b) trained bidirectional character-based lstm-based models to demonstrate the effectiveness of the crosslingual transfer. they have trained bilingual models for languages from romance, slavic germanic, and uralic families. gupta et al. (2020) trained various sequence labelling models for sanskrit. nguyen et al. (2021) trained transformer-based models for various nlp tasks such as pos tagging, morphological feature tagging, and dependency parsing for over 100 languages. nair et al. (2021) carried out a comparative study of existing morphological analyzers for indian languages to conclude that although morphological analyzers exist for indian languages like sanskrit and malayalam, they are not accurate as compared to the high resource baselines. elsner (2021) probed an analogical memory-based framework for oneshot morphological transfer to study the abstract representational concepts learned by the transfer networks. ## dataset challenges creating a multilingual morphology analyzer would require a union of the sets of features across all the languages and all the parts of speech. the morphological features are modeled as categorical variables in fixed output space. the modeling difficulties arise primarily due to the following: (1) absence of feature annotations for indian languages, (2) lack of data for all the parts of speech (pos) except verbs and nouns and (3) variance of markings across pos and languages. the dataset only contains data for verbs and nouns, which restricts our analysis to those pos. for these pos, the feature data is primarily available for gender, number, and person compared to other features, so we carry out transfer analysis for only those features. we provide a summary of annotated data available in we have used various data sources to demonstrate the scalability of the morphology analyzer to 6 indian languages. for languages hindi, telugu, kannada, and bengali, we have used the unimorph 3.0 @xcite dataset. the number of examples varies across languages. for bengali, the number of examples available is 4443; for kannada, it is around 6400; for hindi, there are about 54k examples, while telugu has about 1500 examples. all the examples in the unimorph dataset are either verbs or nouns. for tamil, morphologically annotated data from the tamil dependency treebank @xcite was used. the number of annotated words (verbs+nouns) in the tree bank is 9521, all of which were used. for marathi, we used the dataset from @xcite . the dataset consists of around 21k annotated words, out of which we used 15k words, nouns, or verbs, to have consistency with other datasets. although there are other sources of data, such as @xcite , we stick to the uni-morph dataset wherever possible to ensure higher annotation accuracy. the scope of our work limits demonstrating the usefulness of cross-lingual transfer for morphological analysis, so dataset selection and optimizing the number of examples for creating the best morphology analyzer remains a challenge for future research. ## analysis in this section, we provide further analysis of the cross-lingual transfer of msd tags for verbs and nouns and root word extraction. ## conclusions in this paper, we tested the multilinguality hypothesis for root word extraction and morphosyntactic descriptors (msd) tagging. we trained multilingual models for msd tagging and root word extraction using data of six indian languages spanning two families of the indian subcontinent. we demonstrated the effectiveness of data from languages of the same and different families and how it can be leveraged to train morphological analysis models for low resource languages. we also analyzed how cross-lingual transfer of morphological knowledge happens for nouns and verbs along with the copy bias, which forms a significant component of the root word extraction. our framework can be extended to multiple tags as well as more low resources languages as annotated data becomes available. we see our work as an important step in the direction of bias-aware machine translation to morphologically rich languages.