Adapting transformer models to morphological tagging of two highly inflectional languages: a case study on A ncient G reek and L atin (2024)

Abstract: Natural language processing for Greek and Latin, inflectional languages with small corpora, requires special techniques. For morphological tagging, transformer models show promising potential, but the best approach to use these models is unclear. For both languages, this paper examines the impact of using morphological lexica, training different model types (a single model with a combined feature tag, multiple models for separate features, and a multi-task model for all features), and adding linguistic constraints. We find that, although simply fine-tuning transformers to predict a monolithic tag may already yield decent results, each of these adaptations can further improve tagging accuracy.

## introduction morphological information is an essential enrichment for corpora of highly inflectional languages such as ancient greek and latin. yet given that the field of natural language processing has traditionally been heavily oriented to modern english, a relatively analytic language, the automated processing of morphologically rich languages has been a challenge for some time already (see e.g. @xcite . for ancient greek (henceforth simply 'greek') and @xcite have noted that, as for many other languages, the transformerbased approach has recently become popular for morphological tagging, showing promising results. however, it is still an open question what the most appropriate way is to employ transformer models for this task, i.e. whether specific adaptations are necessary for inflectional languages. 1 for example, for each type (unique word form) in the gum english universal dependencies treebank (see https://universaldependencies.org/ ) there are 10.7 tokens. for the latin proiel treebank there are only 6.5, and for the aim of this paper is therefore to systematically compare a number of adaptations that were previously found to be beneficial for morphological tagging of greek and latin using older methods and assess the importance of these adaptations in a transformer context. we will first discuss previous work related to this topic (section 2). next, we will present the experimental set-up of this project (3), including the data and models we used, and assess which parameter combinations contribute to optimal performance for the two languages (4.1). we will also give a general evaluation of the errors of the best-performing models (4.2). finally, we will summarize the main results of this study and discuss ways for further improvement (5), and address its limitations (6). ## previous work given the vast body of literature on morphological tagging, this section will focus on related work to the central topic of this paper, viz. transformerbased approaches to greek and latin morphological tagging, as well as earlier approaches that have explicitly aimed to adapt tagging techniques to the typological characteristics of these languages. we will therefore not discuss studies that focus on comparing a number of readily available tagging tools (e.g. @xcite , since these tools typically differ on various parameters, so that it is difficult to tell why exactly certain tools are better to handle greek and latin than others. the morphological richness of greek and latin has various consequences: data sparsity arises due to a high number of tokens compared to types, 1 the tag set (i.e. the number of possible combinations of morphological features) is very large and morphology and syntax are often interrelated (e.g. with case marking). as for data sparsity, @xcite advocates for the use of morphological dictionaries for inflectional languages in general, viz. knowledge bases containing lists of morphologically inflected forms and their analysis. in this way the correct analysis for unattested or lowly attested forms can be retrieved from this dictionary instead of solely relying on the training data of the tagger (additionally, even if multiple analyses are present in the lexicon for a given form, the number of possible tags will be heavily constrained by it). various researchers have observed a positive effect of employing such lexica for greek (e.g. @xcite and latin (e.g. @xcite . as for the size of the tag set, it is important to remark that it is only large if we treat the combination of part-of-speech and all the morphological features as one singular label (as is customary for english), i.e. the tag would be 'noun, singular, feminine, dative'. some researchers on inflectional languages have recommended 'splitting' the tags, i.e. making separate predictions for all the individual morphological features, instead (e.g. @xcite . such an approach has been advocated by e.g. keersmaekers (2020), @xcite for @xcite for latin, but so far it has not been compared to a 'singular label'-approach yet. finally, as for the interrelatedness of morphology and syntax, some scholars (e.g. @xcite have shown that performing morphological tagging and syntactic parsing jointly can help both tasks, but since this requires a high performing syntactic parsing model as well, such an approach falls outside the scope of this paper. as noted in the introduction of this paper, recently (encoder-only) transformer models have become popular for greek and latin morphological tagging. they have been employed in various ways, including directly finetuning a pretrained large language model (llm) for this task @xcite @xcite , by extracting the embeddings of a pretrained 2 although sometimes a modern language model is finetuned for ancient languages, as e.g. in @xcite llm and processing them combined with other information through a simpler architecture @xcite @xcite , or, occasionally, utilizing prompts on generative transformer architectures @xcite . the effect of the various parameters described above, including the use of a morphological lexicon and the 'splitting' of morphological tags, has so far not been systematically investigated in a transformer context. in fact, there are reasons to suspect that their effect may be diminished, given that transformer architectures have specific adaptations to handle data sparsity and morphological richness. firstly, transformer models are typically pre-trained on millions (or billions in the case of modern languages) of unannotated tokens, allowing them to recognize forms beyond the specific training set for morphological tagging. nevertheless, the problem remains that morphological richness inherently implies a proportionally larger number of word form types, and due to the closed nature of historical language corpora these pre-trained models are also typically trained on lower amounts of data as compared to modern languages. 2 secondly, in most modern transformer architectures subword tokenization is typically employed (see e.g. @xcite , which splits morphologically complex words in several parts, based on statistical pattern recognition. for example, the tokenizer of the transformer model we will employ for latin (see 3.1) splits the morphologically complex verb honorificentur into honorific+entur, so that even if the full form honorificentur might be scarcely attested, the individual parts honorificand -entur would be more frequent. in this paper we will therefore systematically investigate whether modern transformer architectures have completely superseded the need for any special adaptations for inflectional languages, or if morphological lexica and splitting tags may still offer improvements. ## conclusions the aim of this paper was to investigate whether transformer models need special adaptations to morphologically tag highly inflectional languages with data sparsity, using ancient greek and latin as a test case. we show that, although the most simple approach -i.e. finetuning a transformer model on tags containing all morphological information -already performs decently, special adaptations tailored to the typological nature of these languages can still further improve tagging accuracy. firstly, the use of a morphological lexicon had a clear positive effect on greek tagging accuracy. on latin, conversely, the effect was negative in most cases. this can largely be explained by the quality of the respective lexica: the latin lexicon contained a relatively large number of cases (primarily proper nouns) where not all possible analyses for a given token were recorded in the lexicon, and therefore introduced new tagging errors. nevertheless, the proportion of errors that the latin lexicon corrected (84/881, or about 10%) was still relatively modest. there are multiple explanations why a morphological lexicon might be less necessary than for greek: this might be because greek could be morphologically more complex, or because the pretrained transformer for latin was trained on much more data than for greek, or because the latin data was simply more homogeneous. training separate models for each individual morphological feature had a positive, although very modest effect for both languages. surprisingly, however, multi-task learning did not further improve the results, but had a (slight) detrimental effect instead. we were not able to explain why this was the case. in the future, however, we plan to experiment with other methods to combine the outputs of the individual feature models, as described in @xcite . as for constraining the tag outcomes to the ones occurring in the training data, this had a very slight positive effect for greek and no effect for latin. further adding a linguistically-based list of possible tags did have a slight negative effect for both languages, however. this was caused by a too permissive list of combinatory possibilities, so that feature combinations were predicted that could not co-occur. this is therefore a consequence of the quality of the concrete external list we used, and since it is only through such a list that feature combinations can be predicted that do not occur in the training data, we still generally recommend using this technique. an error analysis revealed where there was room for further improvement. for both languages, data errors and consistency issues made up a very large proportion of errors. most improvement can therefore not be made through more sophisticated machine learning algorithms, but by simply improving the quality of the data. some other errors (e.g. related to logical inferencing or co-references across sentences) would also be hard to solve by the current generation of nlp techniques. a more promising category of errors were related to codependence of morphological and syntactic analysis. in this case, joint syntactic parsing and tagging may offer a possible solution. finally, we should note that, while this paper focused on greek and latin, the techniques we explore are not solely tied to these historical languages, given that there are many other inflectional languages with sparse datasets. we therefore hope that the solutions offered here could also inspire researchers working on similar languages. for the sake of reproducibility and to allow other researchers to make use of the resources this study produced, all the code and datasets we used can be found on github (see 'supplementary material').