Part-of-speech tagging of S wedish texts in the neural era (2021)

Abstract: We train and test five open-source taggers, which use different methods, on three Swedish corpora, which are of comparable size but use different tagsets. The KB-Bert tagger achieves the highest accuracy for part-of-speech and morphological tagging, while being fast enough for practical use. We also compare the performance across tagsets and across different genres in one of the corpora. We perform manual error analysis and perform a statistical analysis of factors which affect how difficult specific tags are. Finally, we test ensemble methods, showing that a small (but not significant) improvement over the best-performing tagger can be achieved.

## introduction the standard approach to automatic part-ofspeech tagging for swedish has been using the hunpos tagger @xcite , trained by @xcite on the stockholm-umeå corpus @xcite . just over a decade later neural methods have reshaped the nlp landscape, and it is time to re-evaluate which taggers are most accurate and effective for swedish text. in this paper we explore part-of-speech and morphological tagging for swedish text. the primary purpose is to see which tagger or taggers to include in the open annotation pipeline sparv 1 @xcite for tagging the multibillion token corpora of språkbanken text, available through korp 2 @xcite . we therefore train and test a set of part-of-speech taggers, which rely on different methods, on a set of corpora of comparable size, with different part-of-speech annotation models. we apply a 5-fold training and evaluation regime. in section 2 we describe the corpora, and in section 3 the taggers and models. we evaluate the taggers along a number of dimensions in section 4, including the potential for using ensemble methods, and discuss the results in section 5. ## taggers we have selected five open-source taggers. our goal was to sample taggers that use different methods, are (or were at some point) known to have high performance and either can be easily incorporated into our annotation pipeline sparv or already are (as hunpos and stanza). this last consideration steers the selection to a large extent (stanza, for instance, has an important advantage of being a convenient pipeline that achieves high performance on other tasks, such as dependency parsing). we also wanted to compare taggers that were state-of-the-art in the "pre-neural" era foot_4 with the current ones. the key properties of the taggers are summarized in as can be seen from the table, different taggers use different kinds of additional information. hunpos does not take any further input. for marmot, we plug in saldo-morphology @xcite , a morphological dictionary of 1m words with a tagset that is similar (but not equivalent) to the suc tagset. from previous experiments we know that using saldo gives marmot a boost when it is applied to texts tagged with the suc tagset (i.e. talbankensbx in our case). we assume it can also boost performance on eukalyptus, since the tagsets are similar, but we do not expect a boost for ud. for stanza, we use word2vec embeddings foot_5 trained on the conll17 corpus @xcite , which was built using the com-moncrawl data and contains approximately 3 billion words for swedish. one of the main ideas of flair is to combine various types of embeddings; the best combination we were able to find was that of the conll17 word2vec and flair's own embeddings (trained on wikipedia/opus foot_6 , size is not reported). for kb-bert foot_7 , we used the bert-base-swedish-cased model, trained by the datalab of the national library of sweden (kb) on 3.5 billion words from the library collection. the collection contains predominantly (85%) newspaper texts, but also official reports from authorities, books, magazines, social media and wikipedia. the training and tagging itself was done as in @xcite , using the run ner.py script from the huggingface framework foot_8 . for stanza and flair, we experimented with using different classic and contextualized embeddings, for instance, word2vec trained on a press corpus @xcite or bert instead of flair's own embeddings, but the results were always slightly worse than those we report. ## evaluation we evaluate the taggers on the treebanks along several dimensions. in the following we report tagger speed and accuracy. we also explore unseen words, specific tags that seem more difficult to get right, as well as an ensemble approach. ## conclusions we applied five taggers to three important swedish corpora. the corpora are of comparable size and have different tagsets. two of them consist of virtually the same texts, but are not entirely parallel. we show that the three neural taggers outperform the two pre-neural (hmm and crf) ones when it comes to tagging quality, but are significantly slower. kb-bert, however, while always yielding the highest accuracy, is also the fastest of the neural taggers, and its speed on gpu is comparable with that of the pre-neural taggers. token-level accuracy of kb-bert (97.2 on average across corpora) is very high, and is decent also for oov tokens (92.5). if we apply sentencelevel accuracy, a less forgiving measure (manning, 2011), we can see that there is actually much room for improvement (67.1). the success of the taggers depends to a large extent on the additional data (embeddings, morphological dictionaries) that they receive as input, of which token embeddings (a.k.a. contextualized or dynamic) seem to be the most powerful ones. it is reasonable to assume that it is also important on which corpus the embeddings were trained. the size of this corpora is comparable for all neural taggers, but kb-bert's is likely to be the most balanced one. the results vary across corpora/tagsets. if we use consecutive splits, talbankensbx always has the highest annotation accuracy and eukalyptus the lowest one. the reason for that is that the two talbankens are more homogeneous (contain only professional prose texts), while eukalyptus contains texts from five different domains, one of which (blogs) is notoriously difficult. the reason for talbankensbx yielding better results than tal-bankenud is probably the less fine-grained tagset, but possibly also more consistent annotation. if, however, we use random splits, the accuracy for eukalyptus goes up, surpassing the one for tal-bankenud. manual error analysis suggests that a high type count, absence of morphological cues, a wide range of syntactic functions, and low frequency make tags more difficult. inconsistent annotation (which is very difficult to avoid in borderline cases) also seems to play an important role. we also perform a statistical analysis of the factors that can potentially affect how difficult the pos+msd tags are. the regression model shows that type-token ratio within tag and average "difficulty" of tokens within tag (measured as entropy of guessing the tag given the token) have con-sistently significant and very strong negative effects on the accuracy. tag frequency has a positive (though not always significant) effect. surprisingly, so does the average "difficulty" of token endings within tag (though the effect is small and not always significant). the results of the statistical analysis partly support the predictions done on the basis of the manual one. in general, this is a promising research avenue which deserves more systematic attention. finally, we test whether the tagger outputs can be combined using ensemble methods, since in theory, there clearly is a potential for that. in practice, it turns out that using a naive bayes classifier it is possible to achieve a very small improvement over the best-performing tagger, but the difference is not statistically significant. the data and scripts that are necessary to reproduce the regression analysis and the ensemble methods are available as supplementary materials 13 .