Leveraging Transformer-Based Models for Predicting Inflection Classes of Words in an Endangered S ami Language (2024)

Abstract: This paper presents a methodology for training a transformer-based model to classify lexical and morphosyntactic features of Skolt Sami, an endangered Uralic language characterized by complex morphology. The goal of our approach is to create an effective system for understanding and analyzing Skolt Sami, given the limited data availability and linguistic intricacies inherent to the language. Our end-to-end pipeline includes data extraction, augmentation, and training a transformer-based model capable of predicting inflection classes. The motivation behind this work is to support language preservation and revitalization efforts for minority languages like Skolt Sami. Accurate classification not only helps improve the state of Finite-State Transducers (FSTs) by providing greater lexical coverage but also contributes to systematic linguistic documentation for researchers working with newly discovered words from literature and native speakers. Our model achieves an average weighted F1 score of 1.00 for POS classification and 0.81 for inflection class classification. The trained model and code will be released publicly to facilitate future research in endangered NLP.

## introduction skolt sami is a minority language in the uralic family, spoken primarily in finland, and is characterized by complex morphosyntactic properties and rich morphological forms (see @xcite . minority languages like skolt sami face significant challenges in the field of natural language processing (nlp) due to their endangered nature, including a lack of extensive annotated datasets and linguistic resources. this scarcity complicates the development of computational models capable of effectively understanding and analyzing the language. moreover, the morphology of skolt sami is highly intricate, with numerous inflections and derivations that present additional challenges for automated processing @xcite . despite these challenges, developing nlp models for minority languages is essential to preserve linguistic diversity and support language revitalization. accurate part-of-speech (pos) and inflection class classification are fundamental steps in understanding the grammatical and semantic structure of a language. such classifications enable downstream nlp applications like machine translation, morphological analysis, and syntactic parsing, which are particularly important for languages with rich morphology. additionally, effective classifiers can assist in improving the current state of fsts by providing greater lexical coverage, ultimately enhancing their ability to handle the full range of morphological variations found in skolt sami. classifiers can also aid researchers in systematically documenting new words collected from literature and native speakers, which is crucial for tracking linguistic evolution in endangered contexts. for skolt sami, pos and inflection class classification can contribute to building digital resources and educational tools, making the language more accessible to both linguists and speakers. to address these challenges, we propose a transformer-based model designed to automate the analysis of skolt sami, specifically for pos and inflection class classification. our approach includes data extraction, preprocessing, augmentation, model training and evaluation. we employed advanced transformer architectures to learn the linguistic features of skolt sami effectively. additionally, we provide both the trained model and the accompanying code publicly to support future research efforts on endangered languages foot_0 . the contributions of this work are as follows: we employed data augmentation techniques, including the generation of morphological forms, to mitigate data scarcity and improve model robustness. ## related work skolt sami has received a moderate amount of nlp research interest as a result of dr jack rueter's amazing work on building the fundamental nlp building blocks for skolt sami foot_1 as a result, skolt sami has an fst @xcite an online dictionary (see @xcite ), a universal dependencies treebank @xcite and some neural models to identify cognates @xcite ). an empirical study by @xcite reveals that the transformer's performance on character-level transduction tasks, such as morphological inflection generation, is significantly influenced by batch size, unlike in recurrent models. by optimizing batch size and introducing feature-guided transduction techniques, the transformer can outperform rnn-based models, achieving state-of-the-art results on tasks such as grapheme-to-phoneme conversion, transliteration, and morphological inflection. this study demonstrates that, with appropriate modifications, transformers are highly effective for character-level tasks as well. recent research @xcite has introduced a joint morphological tagger specifically designed for low-resource agglutinative languages. by leveraging multi-dimensional contextual features of agglutinative words and employing joint training, the proposed model mitigates the error propagation typically seen in part-of-speech tagging while enhancing the interaction between part-of-speech and morphological labels. furthermore, the model predicts part-of-speech and morphological features separately, using a graph convolution network to capture higher-order label interactions. experimental results demonstrate that this approach outperforms existing models, showcasing its effectiveness in low-resource language settings. one notable contribution in this area is a transformer-based inflection system that enhances the standard transformer architecture by incorporating reverse positional encoding and type embeddings proposed by @xcite . to address data scarcity, the model also leverages data augmentation techniques such as data hallucination and lemma copying. the training process is conducted in two stages: initial training on augmented data using standard backpropagation and teacher forcing, followed by further training with a modified version of scheduled sampling, termed student forcing. experimental results demonstrate that this system achieves competitive performance across both small and large data settings, highlighting its efficacy in diverse morphological inflection tasks. recent work @xcite on morphological analysis, generation, and lemmatization for morphologically rich languages has focused on training recurrent neural network (rnn)-based models. a notable contribution in this area is the development of a method for automatically extracting large amounts of training data from finite-state transducers (fsts) for 22 languages, including 17 endangered ones. these neural models are designed to follow the same tagset as the fsts, ensuring compatibility and allowing the neural models to serve as fallback systems when used in conjunction with the fsts. this approach enhances the accessibility and preservation of endangered languages by leveraging both neural and rule-based systems. ## results we conducted six different training experiments to determine the optimal hyperparameter settings for pos and contlex classification. the batch size, embedding size, and hidden layer size were consistent across all experiments, set to 512, 128, and 512 respectively. the following table summarizes the different setups and their corresponding performance metrics for both tasks: the reported results are based on the bestperforming model from these six training experiments. the proposed transformer-based model, when trained on the skolt sami dataset, performed well on both pos and contlex classification tasks. the best-performing model (exp 3) achieved an average weighted f1 score of 1.00 for pos prediction and 0.81 for contlex classification. the hierarchical label encoding strategy and the use of bpe tokenization enabled the model to effectively handle data sparsity and morphological richness. the shared transformer layers provided an efficient way to learn the underlying linguistic structure, while the separate output heads allowed for precise classification for each task. ## discussion and conclusion in this paper, we presented a transformer-based approach for predicting parts of speech and inflection classes (contlexes) for the skolt sami language. the success of the model highlights the potential of combining traditional linguistic tools with modern nlp techniques, particularly for endangered languages. our results demonstrate near-perfect performance for pos classification and reasonably good performance for most contlex categories, although predicting rare categories remains challenging. the results indicate that the use of shared embeddings and multi-task learning can be effective in achieving high accuracy for parts of speech, while data augmentation and careful hyperparameter tuning help in handling the morphological complexities of skolt sami. the observed variability in contlex classification performance, especially for infrequent cate-gories, highlights the challenges of data sparsity and suggests the need for additional efforts in data collection and augmentation. frequent categories like n_sajos benefited from the availability of more examples, whereas rare categories such as n_mainstummush showed lower performance, primarily due to limited training data. this underscores the necessity for expanding the training dataset to cover more diverse lexical entries and reduce biases towards common categories. incorporating additional features, such as syntactic or contextual information, could also enhance the model's understanding of rare categories. the results from limiting the number of words used for prediction suggest that context plays a crucial role in improving model performance. when fewer words were provided to the model, both pos and contlex accuracy suffered, indicating the importance of sufficient contextual information for effective classification. the model showed a consistent improvement in both tasks as more words were added, and the performance eventually stabilized. this demonstrates that using larger contexts allows the transformer model to better capture the linguistic intricacies of skolt sami, improving the reliability of its predictions. moreover, we believe that expanding the dataset to include other related uralic languages could enhance model performance through cross-linguistic transfer learning, benefiting from shared morphological features. another promising direction for future work is the exploration of semi-supervised or unsupervised learning techniques, which could leverage unlabeled data to improve classification performance without relying solely on manually annotated resources. this is particularly relevant given the resource constraints typical for endangered languages like skolt sami. in conclusion, the trained model and code will be released publicly to support future research and application in endangered language processing. we hope that this contribution will aid in the ongoing efforts to preserve and revitalize minority languages by providing computational tools that can be used to automate linguistic analysis, document new lexical entries, and contribute to the development of educational and linguistic resources. future research should continue to focus on enriching the dataset, exploring multi-lingual training, and employing innovative learning paradigms to further advance the field of nlp for endangered languages.