Comparison of Current Approaches to Lemmatization: A Case Study in E stonian (2023)

Abstract: This study evaluates three different lemmatization approaches to Estonian—Generative character-level models, Pattern-based word-level classification models, and rule-based morphological analysis. According to our experiments, a significantly smaller Generative model consistently outperforms the Pattern-based classification model based on EstBERT. Additionally, we observe a relatively small overlap in errors made by all three models, indicating that an ensemble of different approach could lead to improvements.

## introduction recently, two different approaches have been adopted for model-based lemmatization. the generative approach is based on encoder-decoder models and they generate the lemma character by character conditioned on the word form with its relevant context @xcite . the pattern-based approach treats lemmatization as a classification task @xcite , where each class is a transformation rule. when the correct rule is applied to a word-form, it unambiguously transforms the word-form to its lemma. our aim in this paper is to compare the performance of these two lemmatization approaches in estonian. as a third approach, we also adopt the estonian rule-based lemmatizer vabamorf @xcite . as all three approaches rely on different formalisms to lemmatization, we are also interested in the complementarity of these methods. the generative approach is the most flexible, it has the largest search space and therefore it can occasionally result in hallucinating non-existing morphological transformations. on the other hand, the search space of the pattern-based approach is much smaller as the model only has to correctly choose a single transformation class. however, if the required transformation is not present in the set of classes then the model is blocked from making the correct prediction. similarly, the rule-based system can be highly precise but if it encounters a word that is absent from its dictionary the system can be clueless even if this word is morphologically highly regular. one problem with the recently proposed patternbased approach implemented in the udpipe2 is that the transformation rules mix the casing and morphological transformations. this means that for many morphological transformations there will be two rules in the ruleset-one for the lower-cased version of the word and another for the same word with the capital initial letter that needs to be lowered for the lemma-which increases the size of the ruleset considerably and thus artificially complicates the prediction task. thus, in many cases a more optimal approach would be to treat casing separately from the lemmatization. additionally, in the ud estonian treebanks, lemmas include annotations of derivational and compounding processes marked by special symbols. however, these annotations are inconsistent in the data which confuses the models and also complicates the transformation rules. thus, for our evaluation to be unaffected by these factors, we also train our models on the lowercased data with the special symbols removed. lemmatization models are commonly tokenbased, meaning that if the same word-form (with its relevant context) appears several times in the dataset, these repeating instances are kept in the data and thus the training and evaluation sets reflect the natural distribution of words. in contrast, for the morphological reinflection task, the custom has been to train type-based models, in which each lemma and morphological feature combination is presented to the model only once. we were interested in how well the type-based approach can work for lemmatization and thus we also experimented with type-based models where appropriate. in sum, our contributions in this paper are first comparing three different lemmatization approaches on two estonian datasets with different domains with the goal of assessing the complementarity of these systems. secondly, we investigate the effect of casing and special symbols as well as type-vs token-based training and evaluation for each comparison system. ## lemmatization approaches this section gives a brief overview of the current approaches to lemmatization. ## implementation for the generative approach, we adopted the neural transducer by @xcite , previously used for morphological reinflection. neural transducer is a character-level transformer, which takes individual characters of a word form and morphological tags as input, and outputs the resulting lemma characterby-character. for the pattern-based model we adopted an approach similar to udpipe2 @xcite . we used a transformer-based token classification model by fine-tuning @xcite to predict the correct transformation class (@xmath0. our model uses hug-gingface @xcite tokenclassification implementation. moreover, we reuse the code to generate transformation classes from udpipe2. 1 for the rule-based approach, we adopted the estonian rule-based morphological analyzer vabamorf @xcite . we used vabamorf via estnltk, which is a library that provides an api to various estonian language technology tools @xcite . we utilized vabamorf's hmm-based disambiguation capabilities to output a single lemma for each token. ## results tables 2 and 3 show the results on the edt and ewt validation sets respectively. overall, the generative model (in the token-based training setting, see below) outperforms both the patternbased model and the rule-based vabamorf on both datasets. the first column (original) in the top part of the tables 2 and 3 compare the results of the generative model trained on word tokens and word types. additionally, the last column of the in the last row in both tables 2 and 3 show the performance of the vabamorf in the oracle mode, in which case the prediction is considered correct if the true lemma appears in the list of generated candidates. we observe a significant improvement in the accuracy of vabamorf in the oracle mode. this means that a large chunk of errors made by the rule-based approach is the result of poor disambiguation, rather than incorrect morphological analysis. in addition to comparing the performance of different approaches, we are interested in whether there is any complementarity in the errors made by models based on different approaches. figure 2 presents a venn diagram of the token-level errors made by each system. we note that the area of the intersection of all three models is relatively small, meaning that the number of words where all models make an error is quite small, suggesting that different approaches can complement each other in an ensemble setting. ## discussion because the udpipe2's pattern-based approach was highly successful in the sigmorphon 2019 shared task @xcite , we expected it to perform well also in our case, especially because instead of the frozen bert weights used in the udpipe2, we fine-tuned the full model. however, the best shared task lemmatization scores for the estonian language were obtained with the generative contextual lemmatizer by @xcite , which perhaps explains the success of the generative model also in our experiments. when analyzing the errors made by each three approaches, we can see that the set of errors where all models overlap is relatively small (302 out of 5194, 5.8%), which suggests that different approaches can potentially compensate for each other and thus an ensemble of different methods can be useful. the rule-based vabamorf made the largest number of errors. however, when we evaluated it in the oracle mode on edt, it covered the vast majority of correct answers. this implies that vabamorf could gain a lot from a better disambiguator than the current hmm-based one. this was not the case for ewt which, being a web treebank, contains more word forms (such as neologisms, more recent loanwords, and so on) missing from the vabamorf's lexicon. thus, while vabamorf can be a good solution for formal and grammatically correct estonian, it is less suitable for more noisy web texts. the approach to creating transformation rules suggested by the developers of udpipe may output equivalent rules, i.e., when applying these rules to a surface form, the result is identical. we noticed that the pattern-based model is able to identify such cases. this means that an incorrectly predicted label can result in a correct lemma. for example, two rules ↓0;d¦---+m+a and 0;d¦-+@xmath1. the difference between these rules is that the former rule removes three last letters and adds ma-suffix, while the latter removes the last letter, and then replaces the existing letter preceding the existing a with m. we suggest that such a peculiarity may be used to probe language models for morphological knowledge. the five most common rules are shown in ## conclusion we compared three lemmatization approaches on two estonian datasets from different domains and found that on both datasets the generative encoderdecoder approach trained from scratch outperforms both the rule-based vabamorf as well as the patternbased approach fine-tuned from a large pre-trained language model. we observed complementary error patterns for each three approaches, which suggests that ensembling techniques can take advantage of the complementary strengths of each individual approach.