Lemmatization Experiments on Two Low-Resourced Languages: L ow S axon and O ccitan (2023)

Abstract: We present lemmatization experiments on the unstandardized low-resourced languages Low Saxon and Occitan using two machine-learning-based approaches represented by MaChAmp and Stanza. We show different ways to increase training data by leveraging historical corpora, small amounts of gold data and dictionary information, and discuss the usefulness of this additional data. In the results, we find some differences in the performance of the models depending on the language. This variation is likely to be partly due to differences in the corpora we used, such as the amount of internal variation. However, we also observe common tendencies, for instance that sequential models trained only on gold-annotated data often yield the best overall performance and generalize better to unknown tokens.

## introduction lemmatization consists in finding the base form of a given inflected form. the definition of the baseform for a grammatical category can vary across languages. it can include, e.g., finding the masculine singular for an adjective (bèlas 'beautiful.f.pl' > bèu 'beautiful.m.sg), or finding the infinitive for a verb (atten 'eat.3pl.ind.pres' > eaten 'eat.inf'). the main benefit of lemmatization lies in reducing data sparsity by grouping together all surface forms stemming from the same lemma. it is especially useful for morphologically rich languages, for which the high number of surface forms leads to lower token -type ratios. for such languages, lemmatization is systematically used as a preprocessing step for downstream tasks such as parsing, and it is essential for building efficient corpus querying systems. we approach this task from the perspective of two low-resourced, non standardized minority languages: occitan and low saxon. in the case of non standardized varieties, acquiring even minimal amounts of manually lemmatized data can be difficult. one of the reasons is the definition of lemmatization itself: in the absence of a common standard, which approach to lemmatization should be adopted? should lemmatization respect different levels of variation (lexical, morphological, orthographic) which are present in multi-dialect datasets? or should one variety be chosen for lemmatization purposes and used across all dialects? the former solution allows for the preservation of dialectal differences, but limits the positive impact of lemmatization on data sparsity. the latter is more effective in this respect but it compounds lemmatization and normalization, arguably making the task more difficult. furthermore, it can be deemed problematic by the speakers of the language in question. in this paper, we explore both of these approaches: our low saxon dataset adopts an interdialectal lemmatization approach, whereas the occitan dataset's lemmas are dialect-specific. we evaluate the effects of using small, manually annotated datasets for training lemmatization models vs relying on a larger, automatically preannotated corpus. we investigate the utility of developing one general model for all dialects vs training dialectspecific models. since lemmatization typically relies on pos information to aid the processing of ambiguous tokens, we look into different ways of using this annotation layer in our corpora by evaluating two learning paradigms: joint learning and classical, sequential learning for pos-tagging and lemmatization. ## related work lemmatization methods based on machine learning can be divided into edit tree-based approaches and string transduction methods. the edit treebased algorithms (ges; grzegorz chrupala and @xcite derive the sequence of edit operations needed to transform the inflected form into the lemma. the edit tree is used as a label for each wordform -lemma pair. the model learns to predict the edit tree and not the lemma itself, thus treating lemmatization as a classification task. with the advent of neural methods, lemmatization has been recast as a string-transduction task (e.g. @xcite . currently, the main contribution of these approaches to the state of the art seems to be better generalization capacities, measured as the model's ability to correctly lemmatize unseen wordforms. bergmanis and goldwater (2018) report an important improvement on unknown tokens over non-neural approaches, and similar observations are made by @xcite . however, both works remark that the neural networks do not seem to outperform edit tree-based approaches on ambiguous tokens. in general, the capacity to deal with ambiguous tokens is believed to depend on the availability of contextual information, which is supposed to facilitate disambiguation. bergmanis and goldwater (2018) use a sliding character window as context, and @xcite condition the decoder on sentence-level embeddings. these efforts to include contextual information do not seem sufficient to beat the edit-based methods on this type of tokens. the well-suitedness of one type of lemmatization algorithm over the other may also depend on the linguistic properties of a given language. @xcite note that, when evaluating on modern languages, the edit tree-based method outperforms the neural model on both west european and uralic languages, whereas for slavic languages the neural model yields better results. these results would, however, need to be confirmed, since @xcite find that the edit tree-based approach beats the neural model on south slavic languages they investigate. lemmatization is often paired with pos (part of speech) tagging. since inflected forms can be ambiguous as to their lemma, relying on pos-tags can help the disambiguation process. this information can be exploited as part of joint multi-task learning @xcite @xcite or, more traditionally, in a sequential approach, in which the models for two tasks are learned separately, but the lemmatizer relies on the morphological information during training and prediction (e.g. @xcite . @xcite compare lemmatizers for ancient greek based on dictionary lookup that exploit pos information to distinguish ambiguous tokens. alternatively, some approaches do not rely on this type of information at all (e.g. bergmanis and goldwater, 2018), which may simplify lemmatization for low-resource languages. more generally, lemmatization in the lowresource setting has also received attention in recent work. bergmanis and goldwater (2018) evaluate their models both on the full amount of available data and on 10k samples. @xcite explore the lower bound for training data size on indian languages: they compare a standard setting with low-resource settings with only 500 and 100 training instances, in which they rely on data augmentation techniques. @xcite investigate cross-lingual approaches for lemmatizing low-resourced indian languages. in this work, we are particularly interested in the low-resource setting, since the gold standard datasets available for low saxon and occitan are limited in size. we also experiment with the crosslingual and cross-lectal approach by using historical data and related languages. we opt for neural models since we expect a high proportion of unknown tokens in our datasets due to the fact that we are dealing with non standardized languages. we examine both the joint and sequential learning in an attempt to identify the optimal approach to exploit the pos tagging information present in our datasets. ## datasets for both languages, we use two basic datasets: the small dataset is manually annotated and it was available for both languages at the beginning of the experiments reported here. the large datasets are an order of magnitude greater than their small counterparts, but contain only automatic preannotation with pos-tags and lemmas. in the experiments presented here, the small datasets were used for initial training of our models, and we make use of their dev and test splits for training and for evaluation. the large datasets were used as additional training material in various setups in an attempt to improve model accuracies. in the remainder of this section, we provide some quantitative details and descriptions of each dataset. note that the large datasets were not annotated at the beginning of our work. the strategies used to palliate this are described in section 6. in the case of low saxon, both the small and the large dataset stem from the same corpus, described in @xcite , and contain several genres, for instance fiction texts such as fairytales or novels, and non-fiction texts such as letters, announcements or political speeches. the low saxon dataset is roughly split into two time periods: 1800-1939 and 1980-2022. the distribution within the dialect groups is as follows: dutch low saxon 20% and 80%, north german low saxon 87% and 13%, south german low saxon 44% and 56%. for occitan, the small dataset is based on the treebank presented in @xcite and contains predominantly literary texts. the large occitan corpus contains occitan wikipedia articles from 2021, taken from the leipzig corpora collection foot_4 . for both languages, the gold dataset has also been stratified into dialect groups in order to examine the usefulness of dialect-specific training data and evaluate model performance for different dialects. the gold sets are split into train, test and development sets (except in the case of occitan, for which two dialect groups do not have a dev set, the total amount of annotated data being too small) 5 . a quantitative overview of gold splits is given in ## tools we make use of two training paradigms: multi-task learning applied to pos-tagging and lemmatization, in which both tasks are learned as part of the same model, and traditional sequential learning, in which a separate model is trained for each task. we explore the former with machamp (van der goot et al., 2021) and use the stanza nlp pipeline @xcite for the latter. ## strategies for creating large(r) amounts of annotated data one of the dimensions of lemmatization we explored in this work relates to the size and the nature of the training material. specifically, we compared the performance of tools trained on small amounts of gold-annotated data with using larger corpora that were automatically preannotated. as mentioned in section 1, the corpora we used as our large datasets were not annotated at the outset of the experiments presented here. there were, to the best of our knowledge, no freely available models based on neural approaches for the pos-taggging and the lemmatization of low saxon and occitan. the first round of our experiments was therefore dedicated to creating initial models for both tasks which would allow us to produce reliable automatic preannotation. for low saxon, we leveraged an existing historical corpus of middle low saxon to train models that were then transferred to modern low saxon. this had the advantage of using a corpus that was larger than the available gold standard in modern low saxon. for occitan, no comparable historical corpus was available. we therefore relied on bootstrapping using the small dataset. ## discussion and conclusion the overall accuracy results for low saxon are noticeably lower than for occitan, around 10% on average. one possible explanation could be the greater orthographic variation that is likely the reason behind the higher percentages of unknown and ambiguous tokens in low saxon seen in while our occitan corpus makes use of the same spelling convention throughout, the low saxon corpus contains various writing systems even within the same dialect group. furthermore, we trained the models for occitan on major dialects, whereas we used groups of major dialects for low saxon. another reason might be found in the different diachronic structure of the datasets: whereas the occitan data mostly comes from the 20 th and 21 st century, the low saxon dataset covers the period