Utilizing Subword Entities in Character-Level Sequence-to-Sequence Lemmatization Models (2020)

Abstract: In this paper we present a character-level sequence-to-sequence lemmatization model, utilizing several subword features in multiple configurations. In addition to generic n-gram embeddings (using FastText), we experiment with concatenative (stems) and templatic (roots and patterns) morphological subwords. We present several architectures that embed these features directly at the encoder side, or learn them jointly at the decoder side with a multitask learning architecture. The results indicate that using the generic n-gram embeddings (through FastText) outperform the other linguistically-driven subwords. We use Modern Standard Arabic and Egyptian Arabic as test cases, with up to 22% and 13% relative error reduction, respectively, from a strong baseline. An error analysis shows that our best system is even able to handle word/lemma pairs that are both unseen in the training data.

## introduction lemmatization is an important natural language processing task that maps inflected and possibly ambiguous word forms to abstract lemmas. lemmatization is particularly important for modeling morphologically rich languages to address the sparsity of inflected forms. data-driven lemmatization models have the challenge of generalizability beyond the training data, to handle unseen words, unseen lemmas and new contexts. semitic languages, like arabic and hebrew, add other challenges: while words are generally written without disambiguating diacritical marks, unambiguous lemmas are expected to be fully diacritized; and there are many clitics that may be interpreted as part of the stem leading to alternative lemma readings in different contexts (see the results indicate that the generic fasttext embeddings matched or outperformed the other linguistically-driven subwords. these subwords are nevertheless considered linguistically shallow, as opposed to models using deeper -and more expensive -linguistic resources, like morphological analyzers @xcite . we use modern standard arabic (msa) and egyptian arabic (egy) as test cases, where our models provide about 22% relative error reduction for msa, and 13% for egy, from a baseline using character-level sequence-to-sequence model with attention. our models also outperform a strong baseline system that uses rich morphological dictionaries, and can scale to unseen words/lemmas. gold greedy # surface form lemma gloss [lemma] stem pattern root stem pattern root (1a) wllbywt bayt and for the [house]s bywt wll+__w_+ byt bywt wll+_yw_+ bt (1b) albywt bayt the [house]s bywt al+__w_+ byt bywt al+_yw_+ bt (1c) wllsywf sayf and for the [sword]s sywf wll+__w_+ syf sywf wll+_yw_+ sf (2a) alktab kut∼ab the [writer]s ktab al+__a_+ ktb ktab al+__a_+ ktb (2b) alktab kitab the [book]s ktab al+__a_+ ktb ktab al+__a_+ ktb (3a) kbaby kabab my [kebab] kbab __a_+y kbb bab k+_a_+y bb (3b) kbaby bab like my [door] bab k+_a_+y bwb bab k+_a_+y bb ## related work lemmatization has been shown to be very useful in several nlp tasks, including machine translation @xcite and parsing @xcite @xcite , among others. early contributions used finite state machines @xcite , which could not handle unseen words and had limited in-context modeling capacity. other contributions approached lemmatization as a lemma-selection task @xcite , where the goal is to select the correct lemma from a set of lemmas provided by a morphological analyzer. these systems handled in-context analysis better, but had a limited capability of modeling unseen words. to handle unseen words, these models rely entirely on the morphological analyzer, and would usually involve other tasks that might help the disambiguation pipeline (like pos tagging). other data-driven models approach lemmatization as a classification task @xcite , through obtaining the set of edit operations that transform words into lemmas, and then learning to select the optimal lemma from a set of candidates. these candidates are generated by applying all possible edit-trees learned from the training data. there were also several lemmatization contributions utilizing sequence-to-sequence models @xcite , which consider lemmatization as a generation task. several other contributions use additional morphosyntactic features as part of the modeling architecture @xcite . as far as we are aware, no other contributions explicitly utilize other word-level features, like embeddings, stems, or orthographic patterns, in lemmatization. ## approach greedy vs gold stems the stem is the form of the word after all inflectional affixes are removed from the surface form. the surface form can be represented as (prefix+stem+suffix), so stemming removes the affixes at both ends of the word to get its base form. in contrast to lemmatization, which generates proper citation forms. gold stems (and gold patterns and roots) are pre-annotated and can be used while training only, ideally as auxiliary tasks. greedy stems are heuristic, and can be used both at training and runtime. greedy stems are obtained through greedily matching the longest possible affixation sequence from both sides of the surface form, assuming the availability of a set of the potential affixation sequences in the language. the affixes set is inexpensive to obtain through minimal linguistic annotation, or can be obtained directly from the training data if available. the greedily matched sequences are removed from the sides of the word, and the rest is returned as the greedy stem. morfessor base words morfessor @xcite is an unsupervised morphological segmentation tool, based on a probabilistic generative model, designed to handle languages with mainly concatenative morphologies. morfessor generates morpheme-like segmentations, which often resemble linguistic morpheme segments. the morfessor segmentation does not specify the base word explicitly, we therefore consider the least frequent segment in the detected word segments (relative to all the segments in the model) as the base word. this is under the assumption that base words are less frequent than affixes. in msa, the base words from morfessor matched the gold stems about %68 of the time. architecture we use a similar architecture to lematus @xcite ) as a baseline. the model is based on a context-aware character-level sequence-to-sequence architecture, where context is modeled through a sliding window around the target word. lematus uses a fixed 20-character window for each side. however, our experiments show that respecting the word-boundaries, by looking at the characters of the n words before and after the target word, provides better results. we use two lstm layers for both the encoder and decoder (bidirectional for the encoder). we use luong attention @xcite over the encoder outputs h i , and use the last encoder output as the initial state for the decoder. conditioning on the subwords at the encoder we condition on the different subwords through concatenating their embedding vector with the input character embeddings. each character embedding c i is replaced by the concatenation [c i ; w j ] before being fed to the encoder, where w j is the subword embedding for the word in which character i appears in. these features are incorporated at training time and runtime, and do not rely on expensive gold data. this is why we use the greedy stems, patterns, and roots here, instead of the gold subwords. the embedding vectors for the different subwords can be learnt as part of the end-to-end system, or pretrained using a large external corpus. for pretraining, we apply the morfessor and greedy stemming approaches (repeated for the roots and patterns) to a large external corpus, then learn the embeddings using word2vec @xcite . we use the same corpus to learn the fasttext embeddings. multitask learning at the decoder the gold stems, patterns, and roots can also be learnt jointly with the lemmatization task, as auxiliary tasks. in this model the encoder is shared between lemmatization and the other tasks, with the same input and parameters of the bi-lstm network. the different tasks (gold subwords) would have separate decoders specific to each task. the gold subword tasks are also modeled on the character-level, like the lemmatization task. the loss is the average of the individual decoder losses, which are based on minimizing cross entropy h for the decoder of each feature f , where f is the set of word-level features we model (gold stems, roots, or patterns), in addition to lemmatization: ## experiments and results we work with two arabic variants, msa and egy, as test cases. arabic is a morphologically-rich language, with many affixes that result in a large number of inflected forms for each word. datasets we use the words and lemmas from the penn arabic treebank (patb parts 1,2, and 3) @xcite , of about 503k training tokens (of which 63k are for development), for msa, and the arz corpus @xcite , parts 1 through 5, of about 133k training tokens for egy (of which 21k are for development). we follow the data splits recommended by @xcite for both. we use the ldc's gigaword corpus @xcite to train the pretrained msa embeddings, and the bolt arabic forum discussions corpus @xcite for the egy embeddings. baselines we use several baselines to highlight different aspects of the contributions. the first baseline is based on a simple maximum likelihood estimation (mle) approach. we use the training data to identify the most common lemma for each surface form without considering the context, and use this word-lemma mapping for inference. at inference time we return the mapped lemma if the word is seen in the training ## conclusions in this paper we presented a character-level sequence-to-sequence lemmatization model, with access to additional subword features. our experiments show that simply conditioning on the pretrained fasttext word embeddings at the encoder, in addition to the character embeddings, improves accuracy significantly. fasttex embeddings also provide the same or better results compared to sophisticated morphological subword identification approaches. our models outperform a strong baseline, which has access to expensive external resources, and can handle ambiguous and unseen words.