Enhancing Sequence-to-Sequence Neural Lemmatization with External Resources (2021)

Abstract: We propose a novel hybrid approach to lemmatization that enhances the seq2seq neural model with additional lemmas extracted from an external lexicon or a rule-based system. During training, the enhanced lemmatizer learns both to generate lemmas via a sequential decoder and copy the lemma characters from the external candidates supplied during run-time. Our lemmatizer enhanced with candidates extracted from the Apertium morphological analyzer achieves statistically significant improvements compared to baseline models not utilizing additional lemma information, achieves an average accuracy of 97.25% on a set of 23 UD languages, which is 0.55% higher than obtained with the Stanford Stanza model on the same set of languages. We also compare with other methods of integrating external data into lemmatization and show that our enhanced system performs considerably better than a simple lexicon extension method based on the Stanza system, and it achieves complementary improvements w.r.t. the data augmentation method.

## introduction state-of-the-art lemmatization systems are based on attentional sequence-to-sequence neural architectures operating on characters that transform the surface word form into its lemma @xcite . like any other supervised learning model, these systems are dependent on the amount and quality of the existing training data. attempts to develop even more accurate lemmatization systems can focus on improving the model's architecture or obtaining additional data. while annotating additional data is an ongoing process for many smaller languages in the universal dependencies (ud) collection, there are also other data sources available that can be useful for improving lemmatization systems. in particular, we refer to existing rule-based morphological analyzers, lexicons, and other such resources. three potential sources for extracting additional lemma candidates are apertium, unimorph, and ud lexicons initiatives. apertium is an opensource rule-based machine translation platform @xcite . it also includes rule-based morphological analyzers based on finite-state transducers that cover 80 languages. unimorph is a project aimed at collecting annotated morphological inflection data, including lemmas, from wiktionary @xcite , a free open dictionary for many languages. currently, the unimorph project covers 110 languages. ud lexicons is a collection of 53 morphological lexicons in conll-ul format covering 38 languages. ud lexicons mostly use apertium and giellatekno systems to generate the annotations @xcite . several previous works have proposed methods to improve lemmatization systems by augmenting the training data with additional instances @xcite . in this paper, we propose another approach that both modifies the model architecture and leverages additional data. unlike previous work where the model gains from extracting extra knowledge from the additional data provided for training, our primary goal is to teach the model to use external resources, even those that may only be available later during test time. in particular, the proposed system is a dual-encoder model, which receives two inputs for each word: 1) the word form itself to be lemmatized and 2) (optionally) the lemma candidates for that word form extracted from a lexicon or generated by a rule-based system. both inputs are encoded with two different encoders and passed to the decoder. the decoder then learns via two separate attentional mechanisms to generate the lemma via the combination of the regular transduction and by copying characters from the external candidates. this way, the model is trained to use two sources of information-the regular training set and the options proposed by an external resource. the experiments with several models enhanced with external data on 23 ud languages show that the best model using additional lemma candidates generated by the apertium system achieves significantly higher results than the baseline models trained on the ud training set only. also, we compare our method with other methods using external data. the enhanced system performs considerably better than a simple lexicon extension method based on the stanza system, and it achieves complementary improvements w.r.t. the data augmentation method of @xcite . ## related works nowadays, state-of-the-art lemmatization systems are typically based on a neural sequence-tosequence architecture, as demonstrated by the variety of systems presented at the conll 2018 @xcite and @xcite shared tasks. several systems, including the turkunlp pipeline, the winner of the lemmatization track at conll 2018 shared task, use an attention-based translation model @xcite . the input to the system is the character sequence of a surface form (sf), which is "translated" into the lemma by an attention-based decoder. the input sequence can also be extended with pos tags @xcite and morphological features @xcite . another approach was used by the udpipe future system, the second-best model at the conll 2018 shared task. @xcite proposed to produce a lemma by constructing a set of rules that transform the sf into a lemma. these rules can include copying, moving, or deleting a character in the sf, as well as additional rules for changing or preserving the casing. thus, the lemmatization task is rendered into a multi-class classification task of choosing the correct transformation rule among the set of all possible rules generated from the training set. a year later, @xcite improved the result for the lemmatization by adding bert contextual embeddings @xcite to the input, which made them the best lemmatization system at the sigmorphon 2019 shared task. several previous works have proposed to leverage additional data to improve lemmatization. in the simplest form, training data itself can be used to create a lexicon that maps word forms to its lemma. this strategy has been adopted by the stanford neural lemmatization system @xcite , which creates such lexicons from the training sets and resorts to lemma generation only when the lexicon lookup fails. one can easily imagine extending such a lexicon with external resources. @xcite adopted another simple way of using unimorph lexicons to post-fix the morphological features and lemmas predicted by the udpipe system @xcite . the post-fix is performed by simply looking up the sf from the unimorph lexicon and, if the match is found, replacing the model prediction with the tags and lemmas found in the lexicon. another line of work has used additional data to augment the training data set. @xcite augmented their training set by first listing all non-ambiguous word-lemma pairs from unimorph lexicons and then extracted sentences from wikipedia that contained these words. they then trained the context-sensitive lematus model @xcite on this extended partially lemmatized data set. @xcite used apertium's morphological analyzer module to extend the training set for languages with tiny ud datasets. apertium was used to generate all possible morphological analyses to 5000 sentences selected from the wikipedia of the respective language. for each sentence, the most likely analysis sequence was then obtained via a disambiguating language model. the words that were assigned an apertiumgenerated lemma during this process were added to the lemmatizer training set. in the subsequent work, @xcite extended the training data even more. they used apertium to analyze all words found in the conll 2017 web crawl dataset @xcite or in the wikipedia of the respective language. all new words with unambiguous lemma and morphological analysis were added to the augmented training set. ## method the core of the proposed model is the stanford lemmatizer @xcite which is a sequenceto-sequence model with attention. it takes character- h in h cn h in+cn c in+cn output sf pos feats candidate bilstm encoder encoder decoder h in h cn c in c cn lstm attention attention linear bilstm encoder linear linear bilstm encoder linear bilstm encoder bilstm encoder lstm attention attention linear h -hidden state for the last timestep c -cell state for the last timestep h -hidden states from the last layer for all timesteps in -state(s) for the encoded sf-pos-feats cn -state(s) for the encoded candidates level word representation and the pos tag as input and processes them with a bidirectional lstm encoder. then, it passes the encoder outputs to an lstm decoder, which applies a soft dot attention layer after every lstm cell. finally, the output is constructed via greedy decoding. we make several changes to the model architecture as shown in figure 1 . the components comprising the original stanford lemmatizer are marked on the figure with the bold red border. first, we add another encoder that encodes the lemma candidates provided by the external system. the output representations of both encoders are combined with a linear layer and fed to the decoder. secondly, we add another attention layer to the decoder that attends to the outputs of the second encoder. the outputs are finally combined with a linear layer. finally, in addition to the pos tag, we also add morphological features to the first encoder's input. additionally, we implement the encoder dropout to simulate the situation when the external candidates are absent. the value of the encoder dropout that varies in the range of {0.0, 1.0} defines the probability of discarding all candidates from a batch during training. thus, the model will train only the main encoder based on this batch. this helps to train the model to perform more robustly in both situations when the candidates in the second encoder are present or absent. to extract lemmas from the unimorph lexicon, the input surface form (sf) is queried from the lexicon to retrieve the corresponding lemma. some morphological forms in the unimorph lexicons consist of several space-separated tokens; these were discarded. ud lexicons are presented in the conll-ul format, which is an extension of the conll-u format. this makes the extraction process trivial since the lexicons are already pretokenized. for apertium, all generated lemmas were stripped from special annotation symbols, and duplicate lemmas were removed. finally, the simple training set based lexicon solution, similar to @xcite , consists of two lookup dictionaries. the first lexicon maps sf-pos pairs to their lemmas, the second lexicon maps just sf's to their possible lemmas found in the training set. the lemma candidates for a sf are selected by first querying the input sf and pos tag from the sf-pos dictionary and, in case of failure, falling back to the sf dictionary. baselines as the first baseline, we compare our results with stanza, the lemmatization module from the stanza pipeline @xcite , which is a repackaging of the stanford lemmatization system from the conll 2018 shared task @xcite . we used the lemmatization models trained on the udv2.5 available on the stanza web page. as the default baseline, we use our enhanced model, with the second encoder always being empty. ## results the results for be_hse were extreme outliers and were not included in the comparison. the unimorph-enhanced model was excluded from this test as its results did not conform to the normality requirement. as the baseline model performances are already very high and the external information is expected to improve the lemmatization most for the new words unseen during training, we computed the accuracy of the out-of-vocabulary words (oov) for the best performing apertium model and the default baseline. in this context, oov words are those words in the test set that were not seen by the model during training. the results are shown in the right-most section of the ## analysis of the results in this section, we analyze more thoroughly the potential of the proposed method. first, we compare our enhanced system with alternative methods for deploying external data, particularly with the data augmentation method proposed by @xcite and a lexicon extension method implemented based on the stanza system @xcite . secondly, we present more analyses to provide evidence towards the conclusion that the improvements presented for the enhanced model in the previous section can be attributed to our system's ability to make use of external resources supplied to the model via the second encoder. ## conclusion we proposed a method for enhancing neural lemmatization by integrating external input into the model via a second encoder and showed that the system incorporating apertium morphological analyzer significantly improved the performance over the baselines. both @xcite and @xcite used external resources to augment the training data, and thus, the improvement of their system is dependent on the amount and quality of the extended data supplied during training. on the other hand, our method trains the system to use the external information provided during run-time, thus making it independent of the particular external data available during training. we experimentally showed that the enhancing method is both slightly better and complementary to the data augmentation method of @xcite . we also compared our system with a simple lexicon extension method implemented based on the stanza system. when trained and tested in a comparable setting, the proposed enhanced system achieves considerably higher results. although the model's computational complexity is increased by introducing the second encoder, it is counterbalanced by our model being more robust to noise and the ambiguities stemming from the external lexicons. moreover, the main bottleneck in computation originates not from the neural network's increased size but can rather stem from the external system. for example, in our experiments, the main bottleneck in computation originated from executing the transducer-based apertium morphological analyser. to overcome this bottleneck, one possible trade-off between the speed and accuracy is to precompile a candidate list large enough to cover the most frequent words for a given language. this is a problem that also simpler baseline methods adopting external resources have to address. finally, it is worth noting that the proposed method could be beneficial for less-resourced languages. however, establishing this claim would need more systematic experiments exploring specifically on this question, which we did not focus on in this paper. still, because the significant improvements shown in this work are obtained on languages with larger datasets, the possible gains on smaller datasets can be larger.