Analysing cross-lingual transfer in lemmatisation for I ndian languages (2020)

Abstract: Lemmatization aims to reduce the sparse data problem by relating the inflected forms of a word to its dictionary form. However, most of the prior work on this topic has focused on high resource languages. In this paper, we evaluate cross-lingual approaches for low resource languages, especially in the context of morphologically rich Indian languages. We test our model on six languages from two different families and develop linguistic insights into each model’s performance.

## introduction nlp has seen a sharp growth across various frontiers on multiple tasks; for example, today's systems are often required to generate text or to summarize documents. however, a morpheme remains the most basic level of information for most of them @xcite . most of the research in these fields has focused on improving state-of-the-art for high resource languages. in contrast, research on low resource languages has been slow to start. for indian languages, this is a major issue. only some of the 22 scheduled indian languages, which are a subset of the numerous languages spoken and written in india, have enough resources for training a deep learning model. for the remaining languages, the potential for improvement in performance is substantial. most of the current approaches for morphological analysis use cross-lingual transfer learning from a higher resource language to some low resource language @xcite . but choosing the high resource language for transfer learning is still done in an ad hoc manner, with the most common criteria being the phylogenetic distance in the language family @xcite . however, it has been shown that all languages from the same family might not share the same linguistic properties @xcite . in this paper, we use different cross-lingual training methodologies and analyse the resulting source-target language pair performances based on different linguistic factors. ## models we adapt the two-step attention process from the state of the art @xcite on the sigmorphon 2019 morphological inflection task @xcite , switching the input and output to use it as a lemmatiser. the model has four parts: separate encoders for both the tags and the input character sequence, an attention mechanism, and a decoder. the encoder for the lemma is single layer bidirectional lstm. morphological tags are also input to the model for which we use self-attention encoders @xcite without positional embeddings, since the tag embeddings should be order-invariant. at each timestep, on the decoder side, two context vectors are created via two different attention matrices over the output from the encoding of lemma and tag @xcite . the decoder then computes the output in a two-step process: it first creates a tag-informed state by attending over tags using the output from the decoder at the previous time step. we then compute the state vector by attending over the source characters using the tag-informed state. using the updated state, the output character for that timestep is produced. this output is passed through a fully connected layer before applying a softmax to get the output character. we also add structural bias to the attention model that encourages markov assumption over alignments, that is, if the i-th source character is aligned to the j-th target one, alignments from the (i + 1)-th or ith to (j + 1)-th character are preferred. we refer the reader to @xcite for more details regarding the structural bias and @xcite for more details and explanations about the two-step attention process. ## results and discussion ## related work lemmatisation has been tested extensively @xcite , but on datasets that are at least an order of magnitude greater than what we work with. recently, there has been a shift to extremely low resource settings with the sigmorphon 2019 shared task @xcite focusing on cross-lingual learning. however, their task focuses on the reverse direction: given a lemma and a bundle of morphological features, generate a target inflected form. to our knowledge, we are the first ones to study lemmatisation in such a low resource framework. ## conclusion inference-based scripts such as abjad can be difficult for models to learn in extremely low resource scenarios. for other scripts, it is difficult to predict whether cross-lingual models fare better than monolingual models. in general, for a given low resource language, the performance of a language as a source language is a good predictor of gain/loss for other source languages.