How low is too low? A monolingual take on lemmatisation in I ndian languages (2021)

Abstract: Lemmatization aims to reduce the sparse data problem by relating the inflected forms of a word to its dictionary form. Most prior work on ML based lemmatization has focused on high resource languages, where data sets (word forms) are readily available. For languages which have no linguistic work available, especially on morphology or in languages where the computational realization of linguistic rules is complex and cumbersome, machine learning based lemmatizers are the way togo. In this paper, we devote our attention to lemmatisation for low resource, morphologically rich scheduled Indian languages using neural methods. Here, low resource means only a small number of word forms are available. We perform tests to analyse the variance in monolingual models’ performance on varying the corpus size and contextual morphological tag data for training. We show that monolingual approaches with data augmentation can give competitive accuracy even in the low resource setting, which augurs well for NLP in low resource setting.

## introduction natural language processing (nlp) has seen remarkable growth in all its sub-areas like machine translation, summarization, question answering and so on. for all these tasks, though, morphemes remain the most basic form of information @xcite . morpheme identification (lemma and affixes) can assist these very useful large applications by solving the data sparsity problem. good lemmatisers are invaluable tools for handling large vocabulary in morphologically rich languages and thereby boosting performance in downstream tasks, but techniques * these authors contributed equally to this work are limited by resource availability. this is a relevant point for indian languages. for instance, as many as 197 indian languages are in the unesco's atlas of the "world's languages in danger, 2010". even among the 22 scheduled languages of india, there is a wide disparity in resource availability, for example, for konkani and kashmiri @xcite . techniques like porter stemmer are indeed quick solutions, but they are suited only for alphabetic script languages, like english, and not abugida, like bengali @xcite , or abjad, like urdu @xcite , script languages. moreover, creating stemmers requires different language specific stemming algorithms. this requirement of language specific measures comes in the way of scaling the enterprise of creating stemmers for the hundreds and thousands of languages that exist in the world. one might think of ml for stemming-for example, training a neural net with stems and word forms; but almost none of the 22 scheduled indian languages, which is just a subset of the numerous languages spoken and written in india, have resources sufficient for training deep models @xcite . for a majority of indian languages, the absence of dictionaries compounds the problem. most of the current approaches for morphological analysis use the idea of cross-lingual transfer learning from a higher resource language to the low resource language @xcite of interest. we show that even monolingual models can consistently perform with high accuracy with even as little as 500 samples, without cross-lingual training of neural models and without structured information like dictionaries. we further demonstrate good performance in extremely low resource setting with as few as 100 training examples samples to train on and show a competitive performance against cross-lingual models in the same setting. ## related work in @xcite , lemmatisation was performed for small treebanks exploiting the common annotation standard across all languages, and the same task was implicit in @xcite . recently, there has been a shift to extremely low resource settings with the sigmorphon 2019 shared task @xcite focusing on cross-lingual learning. however, their task focuses on the reverse direction: given a lemma and a set of morphological features, generate a target inflected form. ## models a two-step attention process @xcite similar to the sigmor-phon 2019 morphological inflection task @xcite has been adapted for the setup, which consists of four components: encoder for morphological tags, encoder character sequence, attention and a decoder. the inputs to the model are inflected words and morphological tags, and we use selfattention single layer bidirectional lstm without positional embeddings as encoders. at each time step, during decoding, two context vectors are created via two different attention matrices over the output from the encoding of inflected word and morphological tag. at the decoder, we use a two-step process: first we create a tag-informed state by attending over tags using the output from the decoder at the previous time step. second, we use this to attend over the source characters to produce the state vector for the decoder at that time step, which is used for producing the output character for that time step using a fully connected layer followed by a softmax. we also add structural bias to the attention model that encourages markov assumption over alignments, that is, if the i-th source character is aligned to the j-th target one, alignments from the (i + 1)-th or ith to (j + 1)th character are preferred. we refer the reader to @xcite for more details and explanations about the two-step attention process and @xcite for more details regarding structural bias. ## conclusion we have given a methodology for lemmatization of low resource (i.e., availability of small number of word forms) in this paper. for most languages, a monolingual model trained on approximately 1000 training samples gives competitive accuracy, while training on 500 samples gives results at par with rule-based linguistic systems. for extremely-low resource settings as well, monolingual models perform well with the help of data augmentation. even in these scenarios, monolingual models can give competitive results compared to cross-lingual models, a result that is supported by research in other tasks such as morphological inflection @xcite . additionally, in the low resource setting, additional features are an important source of information. even pos tags benefit the training process.