Exploring Neural Architectures And Techniques For Typologically Diverse Morphological Inflection (2020)

Abstract: Morphological inflection in low resource languages is critical to augment existing corpora in Low Resource Languages, which can help develop several applications in these languages with very good social impact. We describe our attention-based encoder-decoder approach that we implement using LSTMs and Transformers as the base units. We also describe the ancillary techniques that we experimented with, such as hallucination, language vector injection, sparsemax loss and adversarial language network alongside our approach to select the related language(s) for training. We present the results we generated on the constrained as well as unconstrained SIGMORPHON 2020 dataset (CITATION). One of the primary goals of our paper was to study the contribution varied components described above towards the performance of our system and perform an analysis on the same.

## introduction morphological inflection is the process of generating varied representations of words based on several linguistic properties @xcite . inflections of words retain their core meanings, however they differ in their word structure. as mentioned by @xcite , morphological inflections can be generated from the root word through two primary methods: concatenative measures and nonconcatenative measures. in the case of concatenative measures, suffixes and prefixes are added to the original word to generate various inflectional forms of the word. non-concatenative inflectional forms are generated by changing the basic structure of the original word. the generation of inflectional forms of a word has proved to be an asset in a wide array of nlp tasks. prominent languages like english, spanish, french, etc. have large corpora that can be utilised to train large scale machine learning applications. however there are several languages in today's world that are not as well documented. these languages are termed as "low resource" languages. morphological inflection has proven to be an effective tool to augment the datasets of "low resource" languages, so that they corpora can be better modeled using nlp techniques. to this end, several studies have been done on morphological inflection on monolingual high resource settings, such as in the @xcite @xcite challenges. however, the low resource setting has been extensively studied in the sigmorphon 2019 and 2020 shared task @xcite . the data in these tasks consists of the form of [i, o, t], where i, o, t stand for input lemma, inflected form and tags respectively. the inflected form is essentially the inflected form of the input lemma upon applying the tags specified by t. in this paper, we present an overview of the various techniques that we implemented to perform the task of morphological inflection in both the constrained and unconstrained settings. we start by describing the different models that we experimented with to improve our performance on this dataset. furthermore, we describe hallucination, sparsemax/sparseloss, adversarial language network and language vector injection techniques that we prototyped to improve the performance of our system. in order to understand the impact of each component on the performance of the system, we perform a detailed analysis on the influence of these techniques on varied set of languages. ## related works in recent years there has been an increase in work in the field of extremely low resource languages. the work recent work done in the field of mor-phological inflection can be divided into two main categories: non-neural approaches and neural approaches. the non-neural based approach proposed by @xcite has two stages, alignment and rule generation. a prominent work that combines neural and non-neural approaches is that of @xcite , where they seek to incorporate monotonicity as an inductive bias in their approach and develop a cubic-time based dynamic programming approach with a greedy decoding scheme. the paper hypothesizes that the monotonic attention-based models perform worse off because they were not jointly trained to incorporate the alignments. neural based approaches have recently outperformed the non neural based approaches. @xcite introduces a neural network based strategy, for the task of morphological inflection generation, for languages that are morphologically rich. the authors introduce an encoder decoder based architecture which makes use of character level embeddings. (c ¸öltekin, 2019) on the other hand adopts the idea from transition-based parsers where the aim is to predict the parsing actions (copy, replace(c), insert(c), delete) in a given state of parser. in the recent years attention based models have gained huge popularity in natural language processing tasks. @xcite introduce a model inspired by sparse sequence to sequence models with a two-headed attention mechanism. the attention and output distributions are computed with sparsemax function and sparsemax loss is optimized. @xcite ) introduce yet another attention based model which is trained on multiple languages and tries to leverage the knowledge learnt on high resource languages for low resource languages. the authors propose a novel two-step attention decoder architecture. moreover, @xcite augment low resource datasets with data hallucination. ## methodology we implemented four variants of sequence to sequence architectures to tackle the problem of morphological injection. we primarily utilize lstm and transformers @xcite to construct our models. additionally we experimented with four techniques hallucination (anastasopoulos and neubig, 2019), sparse max-loss @xcite , language adversarial network @xcite and language vector injection @xcite . ## experimental results we performed our experiments on the data provided in the sigmorphon 2020 shared task. the dataset consists of 90 languages. the data for each language consisted of triplets in the {input, output, tags} format, where the 'output' was the output word generated after applying the morphological tags as specified by 'tags' on the 'input' word. the languages we split into two halves.the first half consisted of 45 languages development languages and the latter half consisted of 45 surprise languages. we made submissions on all 90 languages for two different settings, unconstrained and constrained. for the unconstrained setting we trained our model in a cross-lingual manner. to complement the languages with a low number of training examples we included genetically close languages to augment the training process as explained above. for the constrained setting we restricted our training to only a single language. as explained above we implemented various models such as (lstm), (teld), (tetd) and (tj) augmented with techniques such as (hall), (adv-lang), (sparse), (lvi). since (hall) has proven to perform better than the original setting we augment all languages with less then 10,000 training samples to a complete 10,000 training instances and thus all models and techniques presented below are built on top of hallucinated data. we present the results on a small subset of languages (due to the space constraints) on the development set (since we have results on all the models we trained on the development set) for the unconstrained and constrained settings in we did not experiment with hyperparameters and had a constant set of hyperparameters for all languages. we trained our models with the following hyperparameters 1. a further fine-tuning per language basis might have provided us with a more competitive score. but since one of the primary goals of our study was to understand the influence of the various components on our system we did not pursue this avenue in great detail. we made a total of 5 submissions to the shared task: 3 in the unconstrained settings and 2 in the constrained setting. the submissions made to the unconstrained section are the top 3 ranked results we obtained on the development set and top 2 results for the constrained section. ## analysis our approach of generating morphological inflections, encapsulates several models namely: lstm encoder decoders, transformer encoder lstm decoder(teld), transformer encoder and transformer decoder(tetd) and joint transformers(tj). to supplement these models, we have utilised additional strategies namely adversarial language networks, language vector injection and sparse max and sparse loss. ## future work due to the time constraints we were not able to search through all combinations of the techniques that were mentioned such as lvi+sparse+lang-adv+no-hall and various others. moreover, further fine-tuning the model hyperparameters for each language could have yielded better results. additionally multiple approaches to language vector injection can be explored. the vectors can be fed to the model at every-time step of the encoder by concatenating the input with the vectors or the decoder by concatenating the language vector to the context vector. this form of early injection of the vectors may help the system perform better. another approach can be feeding the language vector to the system in place of @xmath0. the limited availability of supervised data for low resource languages makes it difficult to train the various data hungry neural network models. it has been shown that incorporation of unlabelled data can help improve the performance of such models and thus we propose to integrate a semisupervised approach by learning language models over these low resource languages.these language models inherently contain information about the appropriate character sequences in a given language and thus provide valuable information for predict-ing the next character in the decoding process. we propose to combine the probability generated by the language model with the with probability generated by the inflection model and learn the interpolation weights during training similar to the experimental setup of that of @xcite .the language model can be constructed using a basic recurrent model or even complex models such as bert. @xcite . ## conclusion this paper presents a detailed description of the models that we implemented to undertake the "typologically diverse morphological inflection" shared task. we describe our encoder-decoder based approach using both lstms and transformers. we also describe the different supporting techniques that we implemented, such as hallucination, language vector injection, adversarial language traning and sparsemax. we present a brief subset of the results for the sigmorphon 2020 dataset. we also delve deeper and try to present a detailed analysis of the different components of our model and their influence on the performance.