Annotating N orwegian language varieties on T witter for Part-of-speech (2022)

Abstract: Norwegian Twitter data poses an interesting challenge for Natural Language Processing (NLP) tasks. These texts are difficult for models trained on standardized text in one of the two Norwegian written forms (Bokmål and Nynorsk), as they contain both the typical variation of social media text, as well as a large amount of dialectal variety. In this paper we present a novel Norwegian Twitter dataset annotated with POS-tags. We show that models trained on Universal Dependency (UD) data perform worse when evaluated against this dataset, and that models trained on Bokmål generally perform better than those trained on Nynorsk. We also see that performance on dialectal tweets is comparable to the written standards for some models. Finally we perform a detailed analysis of the errors that models commonly make on this data.

## introduction norwegian twitter data poses an interesting challenge for natural language processing (nlp) tasks. not only do these data represent a set of noisy, usergenerated texts with the kinds of orthographic variation common on social media, but also because there is a considerable number of tweets written in dialectal norwegian. these dialectal variants are quite common and add another level of difficulty for nlp models trained on clean data in one of the two norwegian written forms (bokmål or nynorsk). @xcite compiled a dataset of tweets classified according to whether they are written in primarily bokmål, nynorsk, or a dialect of norwegian. we build upon this work by annotating a subset for part-of-speech (pos). we investigate to what extent available norwegian pos tagging models, that were trained on bokmål and nynorsk universal dependency data @xcite , perform on this twitter dataset. to this end, we use five pos models: three offthe-shelf models, and two developed for the purpose of this work. each of these models was trained on either a dataset of bokmål or nynorsk texts. we explore the performance of each model in terms of accuracy, and investigate which standardized written form can be used as training data and yield good results for non-standardized dialectal texts. the main contributions of this work are: ## background johannessen (1990) outlined a system for automatic morphosyntactic analysis of norwegian nouns in the framework of @xcite . this was among the first systems, if not even the very first, that automatically assigned norwegian texts any morphological information. the first widely used tagger, however, was developed within the taggerprosjektet foot_0 and came to be known as the oslo-bergen tagger foot_1 (obt). rather than continuing and expanding the system of @xcite , obt was implemented in the framework of @xcite . obt was initially a rule-based constraint grammar tagger for norwegian bokmål. later, both support for norwegian nynorsk and a statistical disambiguation component were added @xcite . but one drawback of obt is that it is made for written, edited text, and therefore might not scale well to sources that are not standardised. extending tagger coverage to spoken norwegian dialect transcription, on the other hand, was the objective of both @xcite and @xcite . both sampled data either from the norwegian part of the nordic dialect corpus @xcite ) or the language infrastructure made accessible (lia) corpus. 3 annotations are found in the respective treebanks of the corpora and are accounted for in @xcite and @xcite . besides norwegian, there is a large amount of work on the difficulty of processing noisy data from social media @xcite , including the difficulty of pos tagging on social media @xcite , with dialectal variation @xcite , or whether lexical normalization is helpful @xcite . however, norwegian currently lacks any of these studies. ## data resources for evaluating nlp pipeline tasks for norwegian are scarce. the only dataset available for standard nlp tasks such as pos tagging, lemmatization, and parsing is the norwegian dependency treebank (ndt, solberg (2013), solberg et al. (2014) ) that has been converted to the universal dependencies standard @xcite . there is, however, a notable exception when it comes to transcribed spoken dialectal data, where the lia and ndc treebanks as mentioned above are available with annotations for pos tags, morphological features, lemmas, and dependency-style syntax. despite this, the transcribed texts in the lia and ndc corpora do not share the same characteristics as the twitter data. twitter contains spelling errors and emoji, 4 along with mentions and hashtags. we observe that although our twitter data contains some characteristics of spoken norwegian, such as subjectless sentences as in 1, which is otherwise within the spelling norms, the spelling conventions differ from those of lia and in lia and ndc, all transcriptions are done according to a norwegian-based semi-phonetic standard @xcite , with strict marking of vowel quantity, palatalization, retroflexion, and more. we see that writers on twitter do not conform to any specific spelling norm when writing in their own or another dialect. this means that although not all dialectal traits from a dialect are faithfully preserved, this still leads to much dialectal variation in the twitter data, as things that could have had a common spelling is spelled according to the author's own preference. especially phonetic differences are often not indicated on twitter. because of this, we needed a separate dataset that could be used to evaluate how various systems for norwegian pos-tagging work on dialectal text as it is found on real data from social media platforms. we sampled a balanced subset of the dataset introduced by @xcite , who developed to develop a dialect classifier for norwegian tweets, with the aim to be able to further investigate issues related to dealing with dialectal data on twitter. this subset includes a selection of 38 tweets in bokmål, 31 tweets in nynorsk, and 35 in dialects, which comprises their full test set. we acknowledge that the size of the dataset is small. the postagged dataset is subject to restrictions due to it containing personal information, but is available upon request. ## experiments we test several models trained on available norwegian ud datasets on our twitter data. specifically, we compare obt, stanza, udpipe 2.0, a simple bilstm model, as well as training our own spacy models. both stanza @xcite and udpipe 2.0 @xcite use a bilstm which takes features from 1) pre-trained word embeddings, 2) a trainable frequent word embedding that is randomly initialized before training, and 3) character-level lstm features. while udpipe only uses a softmax layer for classification, stanza instead uses a biaffine classifier to ensure consistency between the upos and xpos predictions. the bilstm model we use is a simplified version of the models used in udpipe and stanza. the model does not take any pre-trained word embeddings as features, but rather uses the vocabulary of the dataset it is trained on to create the embeddings. the model uses a linear layer for classification. the spacy models are newly trained during the present work, and will be released publicly in the near future. since spacy is a fully configurable and trainable pipeline, we used the norwegian bert model described in @xcite with a shared embedding layer for a tagger, morphologizer, and trainable lemmatizer in an effort to optimize the tagger task. ## results and discussion finally, on the dialectal twitter data, the spacy bokmål model once again performs best (83.3). again training on the bokmål data generally performs 12.8 pp better than training on nynorsk data. this may be due to the subset of dialectal tweets, as a manual inspection showed a large number of tweets from central and northern dialects, which share more features with bokmål. a larger number of tweets from western and southern dialects could potentially change this. at the same time, however, it seems clear that the spacy bokmål model performs quite well on all the twitter test data (85.8 acc), so it may simply be a stronger model. ## conclusion in this paper, we have introduced the first dataset of norwegian tweets annotated for part-of-speech, that also include the metadata for the language variety of each tweet (bokmål, nynorsk, dialect, or mixed). we tested several pos taggers trained on ud data and show that, for our twitter data, it is generally better to train on the ud bokmål data, even if testing on nynorsk or dialect. our detailed error analysis showed that the models generally have problems with dialectal pronouns and unfamiliar compounds. finally, we release the newly trained spacy models, and make our annotated data available on request, in order to enable the reproduction of our results.