Is Part-of-Speech Tagging a Solved Problem for I celandic? (2023)

Abstract: We train and evaluate four Part-of-Speech tagging models for Icelandic. Three are older models that obtained the highest accuracy for Icelandic when they were introduced. The fourth model is of a type that currently reaches state-of-the-art accuracy. We use the most recent version of the MIM-GOLD training/testing corpus, its newest tagset, and augmentation data to obtain results that are comparable between the various models. We examine the accuracy improvements with each model and analyse the errors produced by our transformer model, which is based on a previously published ConvBERT model. For the set of errors that all the models make, and for which they predict the same tag, we extract a random subset for manual inspection. Extrapolating from this subset, we obtain a lower bound estimate on annotation errors in the corpus as well as on some unsolvable tagging errors. We argue that further tagging accuracy gains for Icelandic can still be obtained by fixing the errors in MIM-GOLD and, furthermore, that it should still be possible to squeeze out some small gains from our transformer model.

## introduction part-of-speech (pos) tagging is a sequential labelling task in which each token, i.e., words, symbols, and punctuation in running text is assigned a morphosyntactic tag. it is an important step for many natural language processing applications. a token is ambiguous when it has more than one possible tag. the source of ambiguity is polysemy in the form of homographs from the same word class, from different word classes, and also within the declension paradigms of the same word. the task, therefore, entails examining the token itself and its context for clues for predicting the correct tag. for the last mentioned type of ambiguity, which is prevalent in icelandic, it is necessary to find another unambiguous token in the context that the target token shows agreement with and use it to determine the correct target tag. over the last two decades, steady progress has been made in pos tagging for icelandic. various taggers have been presented throughout this period that improved on previous state-of-the-art (sota) methods @xcite @xcite @xcite @xcite @xcite @xcite . work on icelandic corpora has also progressed. existing corpora have undergone error correction phases @xcite , and, in some cases, been expanded with new data @xcite . a new larger gold standard corpus for pos tagging, mim-gold @xcite , was created to replace the older standard, the icelandic frequency dictionary (ifd, @xcite , and multiple alterations have been made to the fine-grained icelandic tagset @xcite . all this variability over the years means that previously reported results for pos taggers are not easily comparable. thus, we train and test four data-driven taggers that have been employed for icelandic (see section 3), using the latest version of mim-gold and its underlying tagset, as well as the latest versions of augmentation data (see section 2). we obtain sota tagging accuracy by training and fine-tuning a convbert-base model in a slightly different manner than previously reported by @xcite (see section 3). with the latest tagging method based on the transformer model finally reaching above 97% per-token accuracy for icelandic @xcite @xcite , the generally believed limit of inter-annotator agreement (manning, 2011), we might ask ourselves if pos tagging is now a solved problem for icelandic. indeed, our evaluation results show that the tagging accuracy of our convbert-base model is close to 98% (see the rest of this paper is structured as follows. in sections 2 and 3, we describe the data and the models, respectively, used in our experiments. we present the evaluation results in section 4, and detailed error analysis in section 5. finally, we conclude in section 6. ## data in this section, we describe the data and the tagset used in our work. ## models in this section, we describe the four data-driven pos tagging models we trained and evaluated: the probabilities of the model are estimated from a training corpus using maximum likelihood estimation. assignments of pos tags to tokens is found by optimising the product of lexical probabilities (p(w i |t j )) and contextual probabilities (p(t i |t i-1 , t i-2 )) (where w i and t i are the i th word and tag, respectively). when work on creating a tagger for icelandic started at the turn of the century, five existing data-driven taggers were tested on the ifd corpus @xcite . tnt obtained the highest accuracy and has often been included for comparison in subsequent work. the model is augmented with n-hot vectors created from all the potential lexical features of the word forms from the dmii. abl-tagger was developed against the ifd but was the first tagger to be applied to mim-gold. we evaluated the four models by applying 10-fold cross-validation (cv) using the standard splits in mim-gold (see section 2). the results are shown in figure 1 : distributions of correctly tagged sentences. the legend shows each set's median (mdn) and mean (m). 7 see https://github.com/orvark13/postr/ for training and evaluation scripts, as well as fine-tuned models. figure 2 : the accuracy improvements between the models for the more frequent lexical categories. solid lines are the per-token accuracy for all tags in that category, and dashed lines are the lexical class accuracy, i.e., the tag category is correct but there is some error in the predicted features. errors within the categories diminish as those lines converge. ## error analysis in this section, we, first, present an analysis of the most frequent errors, and, second, the results of our analysis of the different sources of errors. ## conclusions and future work for icelandic pos tagging, we have reached a point where individual error categories no longer stand out and annotation errors in the corpus are more pronounced, as well as inconsistencies stemming from human label variation. clear annotation errors can be corrected in the corpus, and the tagging guidelines and tag categories can be refined to remove some of the inconsistencies. further gains can as well be squeezed out of the transformer model by using a larger model, i.e., convbert-large instead of convbert-base, increasing the vocabulary size, training it on the 2022 version of igc that adds 549 million tokens, and fine-tuning the hyperparameters for the tagging model. yet, on top of the annotator disagreement, there will always be errors because of a lack of information in the context, as well as the scarcity of examples to learn from for the long tail of infrequent tags. for mim-gold, that unsolvable part of the tagging errors seems to amount to less than 2 pp. therefore, with a little more work, we should be able to confidently pass that 98% accuracy goal (when training and evaluating using the whole tagset) envisioned twenty years ago. a good starting point would be to search for and fix those estimated 1,658 annotation errors in mim-gold, which are a subset of the tagging errors that all four models agree on. to conclude, pos tagging for icelandic is very close to being solved!