Rules and neural nets for morphological tagging of N orwegian - Results and challenges (2023)

Abstract: This paper reports on efforts to improve the Oslo-Bergen Tagger for Norwegian morphological tagging. We train two deep neural network-based taggers using the recently introduced Norwegian pre-trained encoder (a BERT model for Norwegian). The first network is a sequence-to-sequence encoder-decoder and the second is a sequence classifier. We test both these configurations in a hybrid system where they combine with the existing rule-based system, and on their own. The sequence-to-sequence system performs better in the hybrid configuration, but the classifier system performs so well that combining it with the rules is actually slightly detrimental to performance.

## introduction the oslo-bergen tagger (obt, @xcite ) is a widely used tool for morphological tagging of norwegian text. it has existed in various incarnations for around 25 years, first as a purely rule-based system and later coupled with a statistical module for disambiguation. in this paper, we report on our recent efforts to bring the system into the age of neural networks. the question that arises is whether combining the neural system with the existing rules will boost accuracy over a purely neural system. we build two kinds of neural net configurations, one encoder-decoder transformer framework (sequence-to-sequence, seq2seq) and one sequence classification (seqclass) approach. we show that there are challenges in combining rules and neural nets due to divergent tokenisations when the seq2seq approach is employed. in the seqclass approach, on the other hand, the neural net performs so well that combining it with the rule-based approach is actually detrimental to performance, showing that rule-based methods are not required in the morphological tagging of a language like norwegian, where a large language model is available @xcite and there is sufficient labeled data for fine-tuning. however, we still believe that the rule-based system can be useful for lemmatisation and compound analysis, which we do not consider here. the structure of the paper is as follows: in section 2 we give some historical background on obt and in section 3 we describe the current status of its rule-based component. section 4 describes the training and evaluation data that we have used in developing the new systems. section 5 then provides the details of how our neural systems were trained. section 6 describes how they were combined with the rule system. section 7 evaluates the performance of each of the neural systems on their own as well as in combination with the rules. section 8 concludes. the oslo-bergen tagger was originally developed between 1996 and 1998 by the tagger project at the university of oslo. rules for morphological and syntactic disambiguation were written in the first version of the constraint grammar framework @xcite , retrospectively called cg1. the input to cg disambiguation rules is multitagged text, i.e., text where each token has been annotated with all possible lexical analyses. hence, the project also developed a lexicon with lemmas and inflected forms (later known as norsk ordbank 1 ) and a combined tokeniser/multitagger. the tagger was developed for both bokmål and nynorsk, the two written varieties of norwegian. in this article, we will only focus on the bokmål version of the tagger, and only on the tokeniser and the morphological disambiguation. on an unseen evaluation corpus with a wide variety of text genres, obt achieved an f1-score of 97.2 @xcite , with a precision of 95.4 and a recall of 99.0. at the time, this was considered acceptable as the tagger was mostly used to annotate written corpora for linguistic research, where a high recall was considered more important than a high precision. since 1998, both the tokeniser and the cg rule interpreter have been changed or modernised several times by different projects @xcite . the latest version has an updated lexicon and a tokeniser written in python which in most cases mirrors the original tokeniser, with the major exception that multiword expressions like blant annet ('among other things' -adverb) have been removed from the lexicon and are now dealt with in the cg module. the cg1 rules have been translated to the more efficient and expressive cg3 format and are used with a rule interpreter made by the visl project at the university of southern denmark. remaining morphological ambiguities and lemma disambiguation are dealt with by a statistical module, implemented as a hidden markov model. this obt+stat system achieved an accuracy of around 96% @xcite . ## the rule-based tokeniser and tagger in this section, we first present some of the main tasks for the tokeniser and multitagger before we give a short description of the constraint grammar module. the multitagger uses a lexicon with all possible lexical readings, where a reading is a combination of a lemma and a morphosyntactic tag chosen from a set of 149 possible analyses. 2 the main principle for tokenisation is to split tokens on blank space or a sentence delimiter like a full stop or a question mark. for each token identified, the original word form is looked up in the lexicon. non-sentence initial capitalised words are identified as proper nouns, while other words that exist in the lexicon are assigned all readings found there. if the word is not found in the lexicon and is not identified as a proper noun, the word is sent to a compound analyser. most unknown words will get an analysis here, as many of them are productively created compounds. some words will still get the tag ukjent ('unknown') from the tokeniser. these words are often dialect words not standardised in the lexicon or foreign words. figure a in the appendix shows how the tokeniser and multitagger deals with the sentence tv-programmet "ut i naturen" begynner kl. 21.15. ('the tv program "ut i naturen" starts at 21.15.'), which has quotation marks, abbreviations, and a time expression. the tokeniser also identifies sentence boundaries using sentence delimiters, a list of known abbreviations and linguistic rules. headlines are identified by rules as well and get their own tag. the constraint grammar module takes tokenised and multitagged text as input and its main task is to reduce the number of readings to ideally one per word. the number of readings left by the multitagger varies a lot. in the test corpus used in this article (which will be further described in section 4) there are on average 2,04 readings per word. after the cg rules are applied, there are on average 1,09 readings left per word. figure b in the appendix shows the output from the cg module in debug mode for the sentence rosa cupcakes hører kanskje med når man skal ha bloggtreff? ('pink cupcakes might be part of a blog meeting?'). readings that have been removed start with ";" and the id numbers of the rules applied are appended to each reading. note that the english loan word cupcakes is not identified in the lexicon or in the compound analyser and has got the tag ukjent 'unknown'. the compound bloggtreff 'blog meeting' was not in the lexicon but has got two readings from the compound analyser. as the examples show, there are both remove rules (remove a reading) and se-lect rules (select a reading). a rule can be very simple, like rule 2430 in figure 1 that says "select the verb infinitive reading if the verb to the left is a modal auxiliary and not in the set of dangerous infinitives (= not likely infinitives)". #:2430 select:2430 (verb inf) if (not 0 farlige-inf) (-1 m-hj-verb) ; figure 1 : simple select rule figure 2 shows an example of a more complex rule with linked context conditions somewhere to the right in the sentence. the rule says: "choose the subjunction reading -if somewhere to the right there is a safe noun or pronoun (stop looking if a word on the way has a reading that is not an adverb, adjective or determinative) -and -if there is a word in the present or past tense after the noun/pronoun (adverbs between are fine)." ## training and evaluation data the training and evaluation corpora that were used in earlier stages of development of the obt system are no longer suitable because the tagset and the tokenisation principles have evolved. instead of bringing this corpus up to date, we chose to use the norwegian dependency treebank @xcite in the development of the latest version of obt. the bokmål part of ndt is around 300 000 tokens and consists of blog text, news text, parliament proceedings and government white papers. a problem that we only later became aware of is that most of the raw text contained in the ndt probably went into the norwegian bert encoder that we use in our machine learning experiment, which may have caused some data leakage, even if the model did not see the tagged text. while the principles for annotation in ndt and obt are close, there are still differences in detail. to ensure that the annotations were as aligned as possible, we ran obt without statistical disambiguation on the pure text of the ndt and compared the output to the ndt annotations. if the ndt analysis was not among the analyses pro-duced by obt, we either corrected the ndt annotation if that was the source of the error, or changed the rules of the obt system if that could easily be done. this process was iterated a few times. the goal was to improve the quality of the training data for the neural component and to align the output of the obt with the ndt as the annotation guidelines were slightly different. also, since the plan was to combine obt with a neural system, ambiguity reduction by obt was not a goal in itself if we thought the ambiguity could be resolved by the neural component. notice that during this period, the whole data set was used for development, which inflates the performance of the rules (and hence the hybrid system we discuss later on) somewhat. but in practice, relatively few changes were made and we did not achieve a full alignment of the annotation guidelines. the performance of the rule-based system by the end of this phase is shown in ## the neural systems recently, a bert (devlin et al., 2018) pre-trained encoder (nb-bert-base) was published by the norwegian national digital library @xcite . this pre-trained encoder for norwegian provides a rich feature set that was previously lacking for the language. furthermore, since the tagged corpus is very small in comparison to the corpus the pre-trained model was trained on, it is important to use the pre-trained model to be able to generalise to unseen data. we use two different neural system configurations that incorporate this encoder. ## combining neural nets and rules to combine the output of a neural tagger with the cg tagger, we need to find the intersection of the tag assignments produced by the two taggers. ideally, we would be able to find such intersections for each individual token separately. however, since the probability of a reading for a particular token depends on the selected readings for all other tokens in the sentence, the only viable option is to consider readings for entire sentences. thus, for each input sentence, we extract the ten most probable tag assignments produced by the network. then, for each reading in this list, ordered by decreasing probability, we go through each token and check whether the tag assigned by the network is also found among those left by the cg disambiguation rules. if it is not found, we skip to the next reading in the list. if it is found, we go on to check the next token, and so on until we reach the end of the sentence, at which point the reading is picked as the selected one for the sentence. when the tokenisations are different, it is not clear what to do. but if the tokens are the same, but the intersection of the sets of possible tags left by the cg system and the neural net is empty, we can default to the most probable reading in the neural net output. figure 5 shows a case where the tokenisation of the seq2seq neural system does not match with that of obt. the neural system has split the initial, unknown proper name at a hyphen, whereas the cg tagger keeps it as one token. since tokenisation is part of a preprocessing step and misalignments in tokenisation is a problem to be solved separately from tag assignment, we simply disregard such sentences in the evaluation. however, it should be noted that this problem is only acute for the seq2seq system, which produced mismatching tokenisation in 205 out of 2003 sentences (10.2%). for the seqclass system, the problem is smaller: 57 out of 2003 sentences (2.8%). neural net: garosu -gil , som betyr [...] cg: garosu-gil , som betyr [...] figure 6 shows the problem of mismatching tags. for the first word, the cg tagger has left five possible analyses, and the neural net has correctly disambiguated to the plural adjective reading. however, obt did not recognise the second word, cupcakes, and has therefore left an ukjent ('unknown') tag while the neural system has no analysis with that tag. notice that the figure only shows the neural system's most probable assignment of tags to the whole sentence. the actual output is a probability distribution over tag assignments, but in this case, no probability was assigned to any tag assignment containing the ukjent tag for cupcakes, which is the only analysis produced by the rule system. neural net: rosa adj fl pos cupcakes subst appell fl mask @xmath2: rosa adj fl pos adj nøyt ub ent pos adj ub m/f ent pos subst appell ubøy subst appell fem be ent cupcakes @xmath3, we default to the most probable analysis generated by the neural net. this is not necessarily the best option: as we will see in system accuracy pure seq2seq 92.71 seq2seq + obt 94.15 pure seqclass 100.0 seqclass + obt 99.99 ## evaluation and error analysis we evaluate both the seq2seq system and the seq-class system on their own and as combined with the rule-based system in the way described in section 6. this yields four different systems. the performance of the four systems is shown in these numbers are only computed over sentences where the tokenisation matches. this means that the seq2seq system, in both its pure and hybrid form, is tested only on sentences where the seq2seq system, the obt tagger and the gold agree on the tokenisation. as we saw in section 6, this means that 10.2% of the test data are left out. it would have been possible to test the pure seq2seq system on the sentences where its tokenisation agrees with the gold, without considering what obt does, but since we want to compare the pure neural system to the hybrid system, we held the evaluation set constant between these two setups. similarly, for the seqclass system, we left out the 2.8% of sentences where either obt or the neural system had tokenisation that does not match the gold for both the pure and the hybrid system. notice that this means the seqclass system is tested on a larger set of sentences than the seq2seq system. overall, we see that the seqclass system performs best and in fact achieves a perfect score. this is of course a rather debatable result, which we will look into in section 7.2. but notice that the 2.8% of sentences with diverging tokenisations are incomparable and therefore not evaluated here, though they could obviously be considered errors of the system. ## conclusion we have presented our efforts to improve the oslo-bergen tagger for norwegian morphological tagging. two neural systems were trained, based on a sequence-to-sequence setup and a sequence classifier setup, both built on top of the norwegian bert model of @xcite . both were tested on their own and in combination with the rule-based obt system. the sequence-tosequence system did not outperform earlier benchmarks on its own, but improved when combined the rules. however, the sequence classification setup was much better and in fact achieved a surprising perfect score on the test set. while we will explore the causes of this, preliminary testing on new data supports the conclusion that the new system makes very few errors, and we will focus on validating this in a more proper evaluation setting. appendix: sample multitagger and cg @xmath4.</@xmath5.>" "kl." subst appell @xmath6.15</@xmath7.15>" "21.15" @xmath8.15" det @xmath9.</@xmath10.>" "@xmath1?" @xmath11