Analogy Models for Neural Word Inflection (2020)

Abstract: Analogy is assumed to be the cognitive mechanism speakers resort to in order to inflect an unknown form of a lexeme based on knowledge of other words in a language. In this process, an analogy is formed between word forms within an inflectional paradigm but also across paradigms. As neural network models for inflection are typically trained only on lemma-target form pairs, we propose three new ways to provide neural models with additional source forms to strengthen analogy-formation, and compare our methods to other approaches in the literature. We show that the proposed methods of providing a Transformer sequence-to-sequence model with additional analogy sources in the input are consistently effective, and improve upon recent state-of-the-art results on 46 languages, particularly in low-resource settings. We also propose a method to combine the analogy-motivated approach with data hallucination or augmentation. We find that the two approaches are complementary to each other and combining the two approaches is especially helpful when the training data is extremely limited.

## introduction morphological tasks such as the task of morphological inflection generation have attracted great research interest in recent years. sigmorphon has organized annual shared tasks on morphological inflection in the past five years @xcite @xcite @xcite . in the typical sigmorphon shared task of morphological inflection, a lemma (citation form) and a morphosyntactic description (msd) consisting of a set of features are provided, and the task is to generate an inflected form for the lemma corresponding to the msd. neural network models have been very successful in handling natural language processing (nlp) problems, and have achieved new state of the arts in almost every area of nlp, including characterlevel sequence to sequence transformation tasks like morphological inflection, especially when there are abundant labeled data @xcite . however, neural network models are usually very data-hungry, and the performance of such models can suffer when labeled data is limited. unfortunately, the large amounts of labeled data needed is not always available and can be difficult to obtain for many languages. as interest has grown in low-resource nlp, several effective strategies to improve the performance of neural models have surfaced, and neural network models have become the dominant approach in low-resource settings as well. such efforts for the morphological inflection task include engineering the neural network architecture to take better advantage of linguistic knowledge @xcite @xcite , designing data hallucination techniques to generate synthetic data based on existing labeled data @xcite @xcite , augmenting the training data by making better use of labeled or unlabeled data @xcite @xcite @xcite , and cross-lingual transfer learning, i.e. using labeled data in related languages to train models for the target language @xcite . ## experiments we evaluate the performance of all the models on the low-resource languages in the sigmorphon 2020 shared task 0 on morphological inflection @xcite . for our experiments, we regard languages with less than 5,000 training examples as low-resource. there are 46 such languages from 17 language groups in the sigmorphon 2020 shared task 0 data. the training and development data sets of the shared task are provided as triples of lemma, target form and target msd, e.g. jump v;pst jumped. the test data is missing the target form, which the morphology inflection model is expected to predict. this dataset contains labeled data for 1 to 3 different parts of speech (poss) depending on the language (nouns, verbs, and adjectives). we follow the same method as @xcite to reconstruct paradigms from the shared task data. detailed statistics about the data for each language, including training data size, poss, paradigm size for each pos, the number of paradigms per pos, and average paradigm completion rate as well as language group information are provided in tables 5 and 6 in the appendix b. the final number of training examples after the 1-source and 2-source transformation of the original training data is also provided in these tables. in this dataset, the development set is usually 1/7 of the original training set size and the test set is usually 2/7 of the original training data size. the sigmorphon 2020 shared task 0 provides 2 types of neural baselines: a transformer architecture applied at the character level @xcite and a bilstm-based sequence-to-sequence architecture with exact hard monotonic attention @xcite . each type of architecture is trained in four different ways with identical hyperparameters: training one model for each language with and without data hallucination, or training one model per language group with and without data hallucination. this results in 8 baseline models: trm-single, trm-hal-single, trm-shared, trm-hal-shared, and mono-single, mono-hal-single, mono-shared, mono-hal-shared. all the baseline models are trained with only lemma as the source form. since we adopt the transformer architecture for morphological inflection, our work focuses on the comparison with the transformer baselines. we use the implementation of the transformer architecture in the fairseq toolkit foot_1 @xcite , and set the hyperparameters equal to the sigmorphon shared task transformer baselines, except that we use beam search rather than greedy search for decoding. details on the hyperparameters and training heuristics used in the current paper is provided in appendix a. we train one model with the fairseq transformer implementation with the same input-output format as the sigmorphon single-language baseline (i.e. trm-single) and the identical hyperparameters as we use for other model experiments. the result is presented in the row named fairseq-trm-single in ## results and discussion overall performance figure 2 provides an overview of the performance by different models. details about the accuracy of each language by each model is provided in performance and data size figure 3 plots the performance of the transformer models with relation to the training data size. the regression lines indicate that reformatting the training data by adding more analogy sources for the model to learn from is essentially an effective data augmentation approach, but as data increases, the lines in plots (a) and (c) cross each other, indicating that this approach may not be necessary when there is abundant training data available. this is true for the data hallucination approach as well, as is shown in plot (b). plot (d) shows that reformatting the data has similar effects as data hallucination, but that reformatting is in general more effective. is the transformer model we trained with the same hyperparameters as our other models with fairseq implementation, for which the input is the same as trm-single. 1-source and 2-source rows present our reproduction results of the models proposed by @xcite . sigmorphon 2020 shared task 0 baseline model results are copied from the published official results. the highest accuracies for each data size range are in boldface and the second highest is italicized. we further break down the languages by training data size, and present the average accuracy for each data size range in ## conclusion we propose three new ways to reformat training data using an analogy mechanism for morphological inflection in low-resource scenarios: leave-1-out, 1-source+1-crosstable, 1-source+2-crosstable. a systematic evaluation of the model performance shows that the proposed methods that provide both intraparadigmatic and interparadigmatic analogy sources (i.e. 1-source+1-crosstable, 1-source+2-crosstable) are effective. in general, providing more analogy sources for the transformer model to learn from is helpful. we further explore whether the data reformatting approach is orthogonal to data hallucination. experimental results show that combining the two approaches is especially helpful when the training data is extremely limited. however, none of the models we evaluated in our experiments show an acrossthe-board advantage with respect to training data size, paradigm completion rate, or language groups, implying that model ensembling or model selection based on the development data is a good choice to achieve the best morphological inflection performance for a diversified collection of languages. this also indicates that morphological inflection generation is complicated, with many orthogonal factors affecting performance.