GliLem : Leveraging GliNER for Contextualized Lemmatization in Estonian (2025)

Abstract: We present GliLem—a novel hybrid lemmatization system for Estonian that enhances the highly accurate rule-based morphological analyzer Vabamorf with an external disambiguation module based on GliNER—an open vocabulary NER model that is able to match text spans with text labels in natural language. We leverage the flexibility of a pre-trained GliNER model to improve the lemmatization accuracy of Vabamorf by 10% compared to its original disambiguation module and achieve an improvement over the token classification-based baseline. To measure the impact of improvements in lemmatization accuracy on the information retrieval downstream task, we first created an information retrieval dataset for Estonian by automatically translating the DBpedia-Entity dataset from English. We benchmark several token normalization approaches, including lemmatization, on the created dataset using the BM25 algorithm. We observe a substantial improvement in IR metrics when using lemmatization over simplistic stemming. The benefits of improving lemma disambiguation accuracy manifest in small but consistent improvement in the IR recall measure, especially in the setting of high k.

## introduction lemmatization plays an important role in natural language processing by reducing words to their base or dictionary forms, known as lemmas. this process is especially crucial for morphologically rich languages such as estonian, where words can exhibit a multitude of inflected forms. effective lemmatization enhances various downstream nlp tasks, including information retrieval based on lexical search and text analysis. although dense vector retrieval is gaining traction in information retrieval, lexical search methods remain highly relevant, particularly in modern hybrid systems. lexical search excels as a first-stage retriever due to its efficiency with inverted indices, and provides reliable exact term matching that dense retrievers may miss @xcite . recent research demonstrates that lexical and dense retrieval are complementary, lexical matching providing a strong foundation for precise word-level matches, while dense retrieval captures semantic relationships and handles vocabulary mismatches. the complementary nature of these approaches has led to state-of-theart hybrid systems that outperform either method alone @xcite . vabamorf @xcite ) is a rulebased morphological analyzer for the estonian language. it provides one or more morphological analysis (including lemma) candidates for each token in a text, where the token can be a word or a punctuation mark. the vabamorf's analyzer functionality aims to generate all possible morphological analysis and lemma candidates for each word, regardless of its context. however, in order to find the appropriate analysis together with the lemma in a particular textual context, the analyzer output needs to be disambiguated. vabamorf employs a built-in hidden markov model (hmm) based disambiguator that can only look at the word's immediate context to rank the analysis candidates by their likelihood scores. thus, despite its high precision in generating lemma candidates, vabamorf's ability to disambiguate these candidates in context is limited due to its weak representational power. previously, @xcite have shown that, when evaluated on the estonian universal dependencies corpus, vabamorf's disambiguation abilities reach to ca 89% for lemmatization. however, when evaluated in the oracle mode, where a prediction is considered correct if the true lemma appears among the candidates, it achieves an accuracy above 99%. 2 this significant difference highlights the limitations of the vabamorf's current disambiguator and underscores the need for improving its disambiguation component. recent methods to neural lemmatization generally follow two approaches: pattern-based token classification @xcite and generative modeling @xcite . the pattern-based approach predicts for each word a transformation pattern that can be used to transform the word token into corresponding lemma. when built on top of contemporary bert-based encoders, the pattern-based lemmatizer makes use of the contextual token representations directly to make the prediction. the generative approach uses a character-based sequenceto-sequence model to generate the lemma conditioned on the word, relying on disambiguated morphological information as context. while both of these approaches have shown good results on estonian @xcite , neither of them is well suited for developing a new disambiguator for vabamorf. first, the pattern-based token classification approach operates with a limited pattern vocabulary extracted from a training set and cannot handle previously unseen patterns that may be output by vabamorf. secondly, the generative model already assumes the presence of disambiguated morphological analyses making the disambiguation problem circular. recently, an open vocabulary model gliner for named entity recognition (ner) was proposed by @xcite which can be used to match arbitrary text labels with input text spans. in the lemmatizer disambiguation setting, the gliner approach can be used to match the transformation patterns extracted from vabamorf analysis candidates to the spans of sub-word tokens making up words in the text, making it suitable for scoring a limited number of lemma candidates for each word. our first aim in this paper is to investigate whether gliner method can be used to disambiguate the vabamorf's lemma candidates. for that, we modify the gliner implementation to predict the transformation patterns of vabamorf's generated lemma candidates, using the estonian universal dependencies corpus @xcite for training. we find that using this approach boosts the disambiguation accuracy from the hmms 89% to 97.7%, significantly narrowing the gap between the disambiguator and the oracle. our second research question examines the impact of the improved lemma disambiguation accuracy on a downstream information retrieval (ir) task. due to the lack of suitable estonian datasets, we first translate the english dbpedia-entity dataset @xcite into estonian, employing the nllb translation model @xcite . we compare the performance of stemming, vabamorf hmmdisambiguated lemmatization, and vabamorf gliner-disambiguated lemmatization in a bm25 retrieval setup. the results indicate ca 10% improvement in retrieval metrics when using vabamorf lemmatization over stemming, with an additional 1% gain achieved through glinerenhanced disambiguation. overall, our contributions in this paper are threefold: ## vabamorf and gliner in this section, we first give an overview of both the estonian morphological analyzer vabamorf and the open-vocabulary ner model gliner. ## adapting gliner for vabamorf lemma disambiguator we observe that the gliner architecture is flexible enough to be used for essentially any kind of token classification task, including part-of-speech tagging and morphological analysis. to be applicable for lemmatization, we adopt the approach proposed by @xcite that expresses each example of @xmath0. each transformation rule comprises the minimal sequence of character-level edits-commonly referred to as a shortest edit script-such as adding, removing, or replacing characters, required to transform a given form into its lemma. the transformation rules are represented simply as string labels, which are then used in token classification. for specific examples of transformation rules refer to while in theory it would be possible to use lemmas directly as "entities" to be scored instead of transformation rules, that would inflate the number of "entity types" to be learned considerably. effectively, each token type would have to have its own lemmatization label. meanwhile, the transformation rules proposed by @xcite are abstract enough to allow for compact representation of similar transformations, and, according to toporkov and agerri (2024a), they offer stronger generalization than alternative approaches to shortest edit script generation. for instance, some common rules, such as "do nothing" and "upper case the first character", are easily applicable to any surface form. the total number of unique transformation rules relevant for estonian is too large to be used as input to glilem. however, we only aim to score and rank the lemma candidates that the rule-based vabamorf outputs for each token in the text. this limits the total number of possible "entities" to be scored by the number of tokens in the text, but usually it is much lower than that, mainly because the "do nothing" rule is the most common by far even in morphologically rich estonian (see for each input token vabamorf outputs at least one morphological analysis (together with lemma). accordingly, to prepare vabamorf outputs for disambiguation, a transformation rule is found for each token and all of its lemma candidates. the set of strings representing the obtained unique transformation rules are given as entities in the glilem input. glilem outputs a list of spans, each accompanied by a proposed matching transformation rule and its score. the obtained rules are then applied to the respective spans to get the lemmas. the overall glilem architecture, i.e., the gliner architecture applied to lemmatization disambiguation, is schematically represented in figure 1 . ## enhancing disambiguation with glilem we implement the glilem for disambiguating lemma candidates generated by vabamorf. to assess the effectiveness of the glilem approach we evaluate the following approaches: ## impact on information retrieval the problem of lemmatization is usually evaluated in isolation, separately from an actual application. ## conclusion this study demonstrates that integrating an external disambiguation model like glilem with a rule-based morphological analyzer can substantially improve the accuracy of lemmatization in estonian. the enhanced lemmatization bridges the accuracy gap caused by the limitations of vabamorf's built-in disambiguator. this proves our initial hypothesis that the main weakness of vabamorf is in fact its inability to correctly select the lemma candidate in context. additionally, we estimated the effect of improved lemmatization accuracy on an information retrieval downstream task. although the precise effect is difficult to estimate due to the noisiness of the original data and additional noise introduced by imperfect machine translation, we observed small consistent improvements in recall, and especially in the setting with a high k, suggesting that improved lemmatization might translate into actual improvements in a hybrid information retrieval setting.