Learning to Lemmatize in the Word Representation Space (2021)

Abstract: Lemmatization is often used with morphologically rich languages to address issues caused by morphological complexity, performed by grammar-based lemmatizers. We propose an alternative for this, in form of a tool that performs lemmatization in the space of word embeddings. Word embeddings as distributed representations natively encode some information about the relationship between base and inflected forms, and we show that it is possible to learn a transformation that approximately maps the embeddings of inflected forms to the embeddings of the corresponding lemmas. This facilitates an alternative processing pipeline that replaces traditional lemmatization with the lemmatizing transformation in downstream processing for any application. We demonstrate the method in the Finnish language, outperforming traditional lemmatizers in example task of document similarity comparison, but the approach is language independent and can be trained for new languages with mild requirements.

## introduction morphologically rich languages (mrls) encode more information (such as case, gender, and tense) into single word units, compared to analytical languages like english. for example, finnish has 15 different word cases for nouns and adjectives. the different cases generate new words from the syntactical point of view, and in combination with plural forms finnish ends up having 30 different word forms for each noun and adjective. a rich morphology results in extremely large vocabulary and hence low frequency for most word forms in corpora of reasonable size, causing problems, e.g., when learning distributed representations -word embeddings -today widely used in most language processing tasks. while embeddings can be trained for mrls using the traditional methods, such as fasttext @xcite , word2vec @xcite and glove @xcite , their quality still leaves a lot to desire. for example, the results on standard word embedding tests are often worse for mrls @xcite . the natural solution for addressing morphological complexity is lemmatizing, often used as preprocessing before analysis. even though lemmatization loses information by completely ignoring the case, it typically improves performance in various language processing tasks. transformers and other flexible language models @xcite , as well as advanced tokenization methods @xcite , may have reduced the need for lemmatization in general, but it still remains vital for mrls for many tasks @xcite @xcite . traditional lemmatization does not, however, resolve all issues caused by rich morphology, especially as part of a pipeline that uses word embeddings. the embeddings themselves are difficult to estimate for mrls and the embedding methods are typically not transparent about their uncertainty. for instance, the lemma itself may be rare in a typical training corpus and hence we may even switch to using a less reliable embedding, without knowing it. @xcite proposed a possible resolution of training the embeddings on a lemmatized corpus, but this prevents the use of high-quality pretrained embeddings available for many languages and may otherwise hurt embedding quality. the standard processing pipeline also requires access to a good lemmatizer, which may not be available for rare languages, and sometimes they do not work ideally for specialized vocabularies (e.g. medical language). the creation of such a lemmatizer often requires expert knowledge of the target language. we propose a novel approach for addressing rich morphology, illustrated in figure 1 . instead of using a traditional lemmatizer to find the lemmas and using the embeddings for those to represent the content, we do the opposite: we start with the embeddings for all original word forms and then perform lemmatization in the embedding space. this is carried out by a neural network that approximately maps the embeddings of inflected forms into the embeddings of the lemmas. we believe that this may provide embeddings that are better for downstream processing tasks compared to the ones available for the lemmas, for instance when the lemma itself is rare since the model is implicitly able to leverage information across multiple words and cases. another advantage of lemmatization in the embedding space is easy integration as part of the standard modeling workflow that often builds on neural networks anyway, instead of requiring a separate lemmatizer. traditional lemmatization is basically a character-level operation, where grammar rules are used to backtrack the basic form that could have generated the inflected form. we, however, consider word inflections as "bias" in the embedding space, so that the embedding for the inflected word combines (in some unknown way) the semantic meaning of the word and the case information. consequently, our formulation resembles conceptually the problem of bias removal widely studied in the word embedding literature @xcite . the task in bias removal is to transform the embeddings of individual words such that unwanted systematic biases related to gender etc. disappear. our approach can be interpreted in this context as a method of removing undesired morphological information while retaining the semantic meaning of the word. we demonstrate the approach on the finnish language, restricting the analysis for nouns and adjectives that often contain the most important content words for tasks like document similarity comparison or information retrieval. we use pretrained fasttext embeddings @xcite that use subword-level information to provide embeddings for all possible word forms and train a model for mapping them for embeddings of the lemmas using on the dataset extracted from wiktionary by @xcite . the approach is, however, directly applicable to other word classes and languages. besides the pretrained embeddings, it requires only access to (a) existing list of pairs of lemmas and inflected words as in our case, (b) dictionary and morphological generator, or (c) existing traditional lemmatizer for the language. for instance, fasttext provides such embeddings for 157 languages, and morphological analyzers or generators exist for most of these. besides the core concept of lemmatizing in the embedding space, our main contributions are in the specification of practical details for learning the lemmatizers. we specify four alternative neural network architectures, define a suitable objective function and quality metric, and propose a novel idempotency regularization technique to prevent the models from doing anything else besides the lemmatization. we evaluate the approach in document comparison, outperforming the standard pipeline using traditional lemmatizers, and demonstrate it additionally in the task of word list generation. an open-source implementation of the method in python is made available at https://github.com/jalagus/ embedding-level-lemmatization. even though we are the first to directly consider the task of transforming embeddings to lemmatize words, the general question of addressing rich morphology in distributed representations has been studied from various perspectives. @xcite studied the effect of morphological complexity for task performance over multiple languages. they showed that morphological complexity correlates with poor performance but that lemmatization helps to cope with the complexity. @xcite showed a similar effect to hold even with more complex language models, at least for the russian language. @xcite , in turn, showed that for mrls we can improve word similarity comparisons by learning word2vec embeddings from a lemmatized corpus, rather than training them on all data and lemmatizing while learning the task model. @xcite studied supervised lemmatization and morphological tagging using bidirectional rnns with character and word-level embeddings in mrls. they showed that a combination of lemma information and morphological tags improve lemmatization and tagging, but may hurt for english. along similar lines, @xcite suggested using wordembedding clustering to improve lemmatization. as we consider lemmatization from the perspective of bias removal, our work relates to methods for removal of gender bias @xcite . in this line of work, the embedding space is assumed to encode gender information in specific dimensions, so that bias can be minimized by removing them. the main difference to our work is that their goal is primarily in removing the bias, whereas we look for embeddings that retain the semantic meaning of the word well and that are good for downstream task performance. ## evaluation of lemmatization in embedding spaces a traditional lemmatizer either returns the true lemma or not, but when operating in the embedding space of continuous vector representations the question of correctness needs more attention. we start by discussing the evaluation before proceeding to explain the approach itself that builds on these insights. first of all, we note that we can use task performance in any downstream task to evaluate the quality -our ultimate goal is in solving the task well, not in learning the embeddings. we will demonstrate this later in the task of document similarity comparison. however, it is highly useful to also have a generic task-independent metric directly measuring the lemmatization accuracy, which can also be used for motivating the objective for training. we want a good word embedding space lemmatizer m (e w ) to simultaneously satisfy two different criteria: the first criterion is intuitive, matching our goal, but we need to decide how to measure the similarity. for high-dimensional spaces, it is not reasonable to expect a perfect recovery of the embedding e w itself, but instead, we should count all embeddings that are close enough as correct. to determine 'close enough', we use a simple definition based on neighborhoods: lemmatization is correct if the closest neighbor for the transformed embedding of a word w is the embedding of its lemma w . we denote by acc lem the accuracy of nearest-neighbor (rank-1) retrieval accuracy for w in the neighborhood of m (e w ), using euclidean distance for similarity. the second criterion is imposed as we want to consider the lemmatization step as a black box for which we can feed in arbitrary words, including those that are already lemmas. the lemmatizer should not alter them in any way. we measure this by an indirect metric of acc idem , which corresponds to the rank-1 retrieval accuracy for w in the neighborhood of m (m (e w )), the output for an embedding e w passed twice through the model. for a more detailed discussion and justification, see section 4.3. together the metrics acc lem and acc idem characterize the general ability of any embedding-space lemmatizer in a modelindependent way; both are based on retrieval accuracy and can be evaluated without additional assumptions besides the distance measure. we will later use them also to motivate our objective function, a differentiable approximation for their weighted combination. denoting an arbitrary inflected form word embedding by e w and the related lemma word embedding by e w , we wish to learn some mapping m (•) such that m (e w |θ) ≈ e w . we do this by assuming a parametric model family, a neural network, and learning its parameters θ based on a collection of (e w , e w ) pairs of pretrained embeddings in a supervised fashion. for simplicity of notation, we omit the parameters and simply write m (e w ) instead of m (e w |θ) for the rest of the paper. we hypothesize that inflected forms lie on a specific subspace of the embedding space (see figure 2 ) and that we can retrieve the lemmatized forms by a simple, but a possibly nonlinear, transformation in the embedding space. this can be interpreted as the removal of "bias" caused by the inflection. we want this mapping to be lightweight so that it can be integrated as part of a task model with a small computational overhead. complex transformations are discouraged also because they would increase the risk of altering the semantic content captured by the embedding. we discuss two alternative ways of lemmatizing in the embedding space. the first approach learns a separate model m c (e w ) for each word case c so that e.g. partitives and genitives are processed with different models. this allows using simple models even if all of the rich morphology was not constrained in low-dimensional subspaces, and also allows reversing the model for morphological generation (see section 7). for the processing of arbitrary words with an unknown case, we can make a function composite of multiple models, so that the output of one model is always fed as input for the next one. for instance, to lemmatize both partitives and genitives we can compute (m p • m g )(e w ) = m partitive (m genitive (e w )), in either order. assuming the models do nothing else besides remove the effect of the particular case, then this composite function performs the same operation as either model alone, depending on the case of the input word. we naturally cannot guarantee the transformations work exactly like this, but will later present a regularization technique that specifically encourages the models to focus only on the case removal and show empirically that such function composition of multiple models works well. the other alternative is learning a single global model m (e w ) that can lemmatize all word forms. ## model validation we validate the approach and the modeling choices (architecture and regularization), using morphologically rich finnish as an example language. we first evaluate the performance in a taskagnostic manner, before demonstrating case examples in the following two sections. data we validate the approach on finnish language, using pretrained embeddings provided by the fasttext library @xcite . the embeddings were trained on common crawl and wikipedia corpora and have dimensionality of @xmath0. the lemmatization models are trained on the data provided by @xcite which contains words extracted from the open dictionary wiktionary. it directly provides pairs of inflected and base forms for words, so we do not need to construct them. for finnish, the dataset contains 1,136,492 word pairs of adjectives and nouns both in singular and plural, resulting in roughly 42,000 word pairs per word case. each row in the dataset is a pair of form (w, w ) which are then transformed to pairs of word embeddings (e w , e w ) using the fasttext library. training we use adamw optimizer @xcite with a learning rate of 0.0002 and a batch size of 32 for training the models in all experiments, but all reasonable stochastic optimization algorithms would work. we separately validated in preliminary tests that running the optimization until convergence of the training objective does not result in overfitting, and hence for the rest of the experiments we used 50 epochs for training to make sure the models are fully converged. in practice, 20-30 epochs were always enough. all experiments shown here are efficient, so that training individual models on consumer-grade 8-core cpu was done in the order of minutes. ## application: document comparison to demonstrate the method in a typical application, we consider the task of document comparison where the lemmas of content words often provide sufficient information on similarity. we use a dataset provided by the finnish national broadcasting company yle 1 containing news articles written in easy-to-read finnish. we created an artificial dataset by splitting news articles into for all methods, we form a representation for the document by computing the mean of the word embeddings for all words in the document and use cosine similarity between these mean embeddings to compare documents. one could alternatively consider richer document representations @xcite @xcite or more accurate similarity metrics @xcite that might improve the overall accuracy, but we chose the most commonly used approach that is easy to understand to focus on demonstrating the effect of the lemmatization. we measure performance by retrieval accuracy, by computing the rank of the second half of a given document amongst the set of all 10,000 second halves. figure 3 shows the overall performance of the different model variants as a func-tion of the regularization parameter, measured by rank-1 accuracy. we observe three clear results: (a) all ways of lemmatization clearly improve the task performance compared to no lemmatization at all, (b) lemmatization in the embedding space using case-specific models is considerably better than the alternatives of traditional lemmatizer and the global model lemmatizing in the embedding space, and (c) idempotency regularization is crucial, but the method is extremely robust with respect to the specific choice of α -all values between 0.2 and 0.9 result in almost identical performance. ## application: word list generation even though our main goal is to learn lemmatizers, we note that that the approach is more general. instead of training a lemmatizer m c (e w ) ≈ e w , we can use the exact same architectures and data for training g c (e w ) ≈ e w to learn generators that provide the embedding for the inflected form for some particular case c. we demonstrate this via the simple application of word list expansion, which could be used simi- ## conclusions for mrls lemmatization helps in many tasks. we showed that the conventional pipeline using traditional lemmatizers as preprocessing can be replaced by lemmatization in the embedding space. already simple neural networks can transform the embeddings of inflected words so that the closest word in the embedding space is of the correct lemma. this verifies lemmatization in the embedding space is possible, but in real applications, we naturally would not convert the result back to the lemma. instead, any downstream task simply processes the lemmatized embeddings directly. we showed that the method outperforms conventional lemmatization preprocessing in the document similarity comparison task, which implies we are not merely learning to replicate the exact lemmatization but instead learn embeddings that