Strategies to Improve Low-Resource Agglutinative Languages Morphological Inflection (2023)

Abstract: Morphological inflection is a crucial task in the field of morphology and is typically considered a sequence transduction task. In recent years, it has received substantial attention from researchers and made significant progress. Models have achieved impressive performance levels for both high- and low-resource languages. However, when the distribution of instances in the training dataset changes, or novel lemma or feature labels are predicted, the model’s accuracy declines. In agglutinative languages, morphological inflection involves phonological phenomena while generating new words, which can alter the syllable patterns at the boundary between the lemma and the suffixes. This paper proposes four strategies for low-resource agglutinative languages to enhance the model’s generalization ability. Firstly, a convolution module extracts syllable-like units from lemmas, allowing the model to learn syllable features. Secondly, the lemma and feature labels are represented separately in the input, and the position encoding of the feature labels is marked so that the model learns the order between suffixes and labels. Thirdly, the model recognizes the common substrings in lemmas through two special characters and copies them into words. Finally, combined with syllable features, we improve the data augmentation method. A series of experiments show that the proposed model in this paper is superior to other baseline models.

## introduction morphological inflection generates a word form given a lemma and target morpho-syntactic descriptions (msds) @xcite . for example, give the word 'dog' and the msd labels 'n;pl', and to generate the word 'dogs'. similar to morphological analysis @xcite and morphological segmentation @xcite , morphological inflection is a fundamental task in natural language processing (nlp). it plays * *:corresponding author a crucial role in various downstream applications such as dependency parsing @xcite , machine translation @xcite @xcite , and others. researchers have shown increasing interest in morphological inflection in recent years, and the research methods have evolved from traditional linguistic knowledge-based finite-state transducers (fsts) to sequence-to-sequence frameworks @xcite . the construction of relevant datasets @xcite and the advancement of research approaches @xcite have significantly reduced the difficulty of morphological inflection, but new challenges have also emerged. the model achieves high accuracy when both the lemma and feature set are attested in the training set. however, when lemma or feature sets are unattested in training, or in cases similar to the "wug test" @xcite , the model's accuracy begins to decline @xcite , even in high-resource languages. because the dataset of low-resource languages is too small, training neural network models can result in label bias, where the model tends to output characters commonly seen in the training set @xcite . it is very effective to augment training data in low-resource with a data hallucination approach @xcite . @xcite proposed a data augmentation based on characters, while @xcite argue that data hallucination based on strings or syllables approach (such as 2-gram, 3-gram, 4-gram, etc.) is more effective than character-based. this is because character-based hallucination breaks the original syllabic structure of words. additionally, in sequence-to-sequence models (seq2seq), the input usually includes both the lemma and msds. when the lemma and msds are lengthy, it cannot be guaranteed that each label will impact every character. in the agglutinative language morpholog-ical task, msd affects the beginning and ending of the word, with very few influences on the internal structure of the word, as shown in the following example in kyrgyz: figure 1 : an example in kyrgyz in the example, the lemma is on the left, and the word is on the right. we divide the word into two parts: red is the stem, and blue is the suffix, and the stem is a part of the lemma. during model predictions, errors can occur not only in the suffix but also in the stem. furthermore, when the lemma is connected to the suffixes, there may be character substitution, insertion, and deletion. there are too many uncertainties regarding which characters undergo each type of transformation @xcite . these uncertainties can also change syllable categories at the connection points. all these problems make low-resource agglutinative language morphological inflection more challenging. therefore, based on the above problems and considering the characteristics of agglutinative language syllables, this paper proposed four strategies to address them. the first strategy aims to reduce the impact of agglutinative language phonetic variations by incorporating a convolution module in the model's encoder. this module extracts syllabic features (like n-grams). the second strategy, inspired by the work of @xcite , adds reversed token embeddings and positional encodings in the encoder's input. additionally, label positions are marked, enabling the model to learn the correspondence between suffixes and labels and the impact of labels on each character. the third strategy aims to alleviate errors in the stem. in the encoder, special characters are added to the beginning and ending of the lemma's stem. in the decoder, each character of the lemma is marked to indicate whether it should be copied. the fourth strategy is to avoid breaking the syllable categories of lemmas and words during data augmentation. letter type (sound: vowel or consonant) is determined when randomly sampling. if the letter being replaced is a vowel, it is substituted with another vowel in the language; a consonant is replaced with another consonant. we evaluate our model on five low-resource agglutinative languages, kazakh, kirgiz, tatar, uyghur, and uzbek, in unimorph. the experiments show that the performance of the model proposed in this paper is superior to that of other comparable models. the baseline model (baseline-neural model) with data hallucination and three strategies have improved the overall accuracy of the model by 9.54% and 4.17%, respectively. in summary, our main contributions are as follows: ## related work in recent years, the development of morphological inflection has significantly been promoted by the sigmorphon shared tasks @xcite @xcite . research on morphological inflection mainly focuses on rule-based (such as fst) @xcite , statistical @xcite , and neural network-based models @xcite @xcite . additionally, data augmentation (anastasopoulos and neubig, 2019; @xcite can also improve the performance of models in low-resource languages. seq2seq models, such as rnn+attention @xcite or transformer @xcite @xcite , have become popular framework for morphological inflection in recent years. the lemma and tags are usually input together in this framework, and the model generates the inflected word. for example, given the input 'dog+n+pl', the output should be 'dogs' @xcite . based on the transformer, @xcite modified position encoding of msds in the input sequence to 0 and added embedding type to distinguish between characters and features. this modification makes the model more suitable for morphological inflection. transformer can achieve high accuracy in high-resource or simple conditions where both the lemma and msd have attested in the training dataset. however, training high-accuracy models in low-resource or complex situations where the lemma or tags are unattested in the training dataset is challenging. through experimental analysis @xcite , it has been found that for some languages, there is a portion of the generated word where the lemma and feature tags correspond to the common strings. therefore, improving the model's ability to copy characters can enhance its performance. singer and kann (2020) proposed a pointer generator transformer, which uses a copying mechanism to generate a character probability distribution. this model achieved a 4.46% improvement over the vanilla transformer in low-resource languages. @xcite proposed a characterlevel neural transducer that operates over traditional edit actions based on their previous work @xcite . they optimized the training procedure using mini-batches and only relied on the teacher-forcing approach, i.e., using gold labels rather than what was predicted during the training phase. morphosyntactic features were treated individually, and their embeddings were summed. @xcite proposed a two-step attention decoding structure and augmented the dataset through data hallucination. firstly, they identified the "stem" (the common part when comparing lemma and word, where there is one or several stems) based on the lemma-word pairs in the dataset. then, they randomly replaced the string in the stem, except for the first and last strings. @xcite suggested that in morphological inflection, only forward distances are usually encoded while ignoring backward distances. therefore, they added reverse positional encoding based on the char-transformer model. firstly, they trained the model using standard backpropagation and teacher forcing based on the data augmentation proposed by @xcite , saving the best model on the validation set. then, they further trained the model using student forcing. finally, this model achieved an accuracy improvement of 9.6% and 8.6% compared to the baseline model in low-resource and highresource scenarios. @xcite proposed two models in the sigmorphon 2022 shared task: a neural network-based model and an fstbased model. the fst model outperformed the neural network-based model in specific languages. this indicates that for endangered languages or lowresource scenarios, data-driven methods are still immature and rely on linguistic rules. although fst models achieve higher accuracy in specific languages, collecting or annotating linguistic rules is costly and time-consuming. thus, building a highperformance model using existing data resources is crucial. therefore, this paper focuses on five low-resource agglutinative languages. based on the baseline model -transformer, four strategies are proposed to improve the model's accuracy and robustness by incorporating morphological features of agglutinative languages. ## approaches in this section, we describe our strategies for the inflection task. ## conclusion this paper addressed the challenges of lowresource agglutinative language morphological inflection and proposed four strategies. firstly, to tackle the main issue of limited training data in lowresource settings, a data hallucination approach that incorporates syllable features is introduced. a syllable feature extraction module is added to the encoder, enabling the model to learn the context and transformation of characters through syllables. secondly, the lemma and msds are separately encoded at the encoder's input. reversed token embeddings and positional encoding are also incorporated to establish correlations between labels and generated suffixes. lastly, the model's ability to copy common parts of lemmas is enhanced by marking common substrings at the encoder-decoder. experimental results demonstrate that the proposed strategies effectively alleviate the issues caused by data scarcity or agglutinative language features, and all strategies lead to improvements in model accuracy, outperforming other comparative models. this paper initially explores the agglutinative language morphological inflection model in lowresource scenarios. in future research, we will continue optimizing the model's ability to learn positional encoding and extract syllable features, further enhancing its generalization capabilities.