N or D ial: A Preliminary Corpus of Written N orwegian Dialect Use (2021)

Abstract: Norway has a large amount of dialectal variation, as well as a general tolerance to its use in the public sphere. There are, however, few available resources to study this variation and its change over time and in more informal areas, on social media. In this paper, we propose a first step to creating a corpus of dialectal variation of written Norwegian. We collect a small corpus of tweets and manually annotate them as Bokmål, Nynorsk, any dialect, or a mix. We further perform preliminary experiments with state-of-the-art models, as well as an analysis of the data to expand this corpus in the future. Finally, we make the annotations available for future work.

## introduction norway has a large tolerance towards dialectal variation @xcite and, as such, one can find examples of dialectal use in many areas of the public sphere, including politics, news media, and social media. although there has been much variation in writing norwegian, since the debut of nynorsk in the 1850's, the acceptance of dialect use in certain settings is relatively new. the official language policy after world war 2 was to include forms belonging to all layers of society into the written norms, and a "dialect wave" has been going on since the 1970's @xcite . from 1980 to 1983 there was an ongoing project called den første lese-og skriveopplaering på dialekt 'the first training in reading and writing in dialect' @xcite , where primary school students were allowed to use their own dialect in school, with tove bull as project leader. bull et al. * the authors have equal contribution. at the same time, there are two official main writing systems, i.e. bokmål and nynorsk, which offer prescriptive rules for how to write the spoken variants. this leads to a situation where people who typically use their dialect when speaking often revert to one of the written standards when writing. however, despite there being only two official writing systems, there is considerable variation within each system, as the result of years of language policies. today we can find both 'radical' and 'conservative' versions of each writing system, where the radical ones try to bridge the gap between the two norms, while the conservative versions attempt to preserve differences. however, it is still natural that these standards have a regularizing effect on the written varieties of people who normally speak their dialect in most situations @xcite . as such, it would be interesting to know to what degree dialect users deviate from these established norms and use dialect traits when writing informal texts, e.g. on social media. this could also provide evidence of the vitality of certain dialectal traits. in this paper, we propose a first step towards creating a corpus of written dialectal norwegian by identifying the best methods to collect, clean, and annotate tweets into bokmål, nynorsk, or dialectal norwegian. we concentrate on geolects, rather than sociolects, as we observe these are easier to collect on twitter, i.e. the traits that identify a geolect are more likely to be written than those that identify a sociolect. this is a necessary simplification, as dialect users rarely write with full phonetic awareness, making it impossible to find dialect traits that lie mainly in the phonology. as such, our corpus relies more on lexical and clear phonetic traits to determine whether a tweet is written in a dialect. we collect a corpus of 1,073 tweets which are manually annotated as bokmål, nynorsk, dialect, or mixed and perform a first set of experiments to classify tweets as containing dialectal traits using state-of-the-art methods. we find that fine-tuning a norwegian bert model (nb-bert) leads to the best results. we perform an analysis of the data to find useful features for searching for tweets in the future, confirming several linguistic observations of common dialectal traits and find that certain dialectal traits (those from trøndelag) are more likely to be written, suggesting that since their traits strongly diverge from bokmål and nynorsk, they are more likely to deviate from the established norms when composing tweets. finally, we release the annotations and dialect prediction models for future research. foot_0 ## related work the importance of incorporating language variation into natural language processing approaches has gained visibility in recent years. the vardial workshop series deals with computational methods and language resources for closely related languages, language varieties, and dialects and have offered shared tasks on language variety identification for romanian, german, uralic languages @xcite , among others. similarly, there have been shared tasks on arabic dialect identification @xcite . to our knowledge, however, there are no available written dialect identification corpora for norwegian. many successful approaches to dialect identification use linear models (e.g. support vector machines, multinomial naive bayes) with word and character n-gram features @xcite , while neural approaches often perform poorly @xcite (see @xcite for a full discussion). more recent uses of pretrained language models based on transformer architectures @xcite , however, have shown promise @xcite . corpus-related work on norwegian dialects has mainly focused on spoken varieties. there are two larger corpora available for norwegian: the newer nordic dialect corpus @xcite , which contains spoken data from several nordic languages, and the language infrastructure made accessible (lia) corpus, which in addition to norwegian also contains sámi language clips. foot_1 there is also the talk of norway corpus (lapponi et al., 2018), which contains transcriptions of parliamentary speeches in many language varieties. while they contain rich dialectal information, this information is not kept in writing, as they are normalized to bokmål and nynorsk. these resources are useful for working with speech technology and questions about norwegian dialects as they are spoken, but they are likely not sufficient to answer research questions about how dialects are expressed when written. the transcriptions in these corpora also differ from written dialect sources in the sense that they are in a way truer representations of the dialects in question. in writing dialect representations tend to focus more on a few core words, even if the actual phonetic realization of certain words could have been marked in writing. ## data collection in this first round of annotations, we search for tweets containing bokmål, nynorsk, and dialect terms (see appendix a), discarding tweets that are shorter than 10 tokens. the terms were collected by gathering frequency bigram lists from the nordic dialect corpus @xcite from the written representation of the dialectal varieties. two native speakers annotated these tweets with four labels: bokmål, nynorsk, dialect, and mixed. the mixed class refers to tweets where there is a clear separation of dialectal and nondialectal texts, e.g. reported speech in bokmål with comments in dialect. this class can be very problematic for our classification task, as the content can be a mix of all the other three classes. we nevertheless keep it, as it still reflects one of the written representations of norwegian. in example 1, we show two phrases from the nordic dialect corpus, from a speaker in ballangen, nordland county. we show it in dialectal bokmål nynorsk dialect mixed total train 348 174 274 52 848 dev 52 20 30 4 106 test 38 31 35 6 110 total 438 225 348 62 1,073 the two annotators doubly annotated a subset of the data in order to assess inter annotator agreement. on a subset of 126 tweets, they achieved a cohen's kappa score of 0.76, which corresponds to substantial agreement. given the strong agreement on this subset, we did not require double annotations for the remaining tweets. certain traits made the annotation difficult. many tweets, especially those written in dialect, are informal, and therefore contain more slang and spelling mistakes. for example, jeg 'i' can be misspelled as eg, which if found in a non-nynorsk setting could indicate dialectal variation. spelling mistakes should not interfere with dialect identification, but as some tweets can contain as little as one token that serve to identify the language variety as dialectal, this can cause problems. some dialects are also quite similar to either bokmål or nynorsk, and speakers might switch between them when speaking or writing. similarly, certain elements can be indicative of either a geolect or a sociolect, e.g. the pronoun dem 'they' as the third person plural subject pronoun (de in bokmål and nynorsk), which in a rural setting might be typical for an east norwegian dialect, while in an urban setting might be a strong sociolectal indicator. tweets with similar problems are annotated in favor of the dialect class. additionally, there is the problem of internal variation. a tweet can belong to a radical or conservative variety of standardized norwegian, e.g. riksmål, and thereby not be dialectal. however, this distinction can be difficult to make if a writer uses forms that are now removed from the main standards (bokmål and nynorsk), and therefore become more marked, such as sprog instead of språk 'language'. ## dialectal traits to find the most salient written dialect traits compared to bokmål and nynorsk, we perform a χ 2 test @xcite on the occurrence of unigrams, bigrams, and trigrams pairwise between bokmål and dialect, and then nynorsk and dialect and set @xmath0.5. the most salient features (see ## experiments we propose baseline experiments on a 80/10/10 split for training, development and testing and use a multinomial naive bayes (mnb) and a linear svm. as features, we use tf-idf word and character (1-5) n-gram features, with a minimum document frequency of 5 for words, and 2 for characters. we use mnb with @xmath1.01, and svm with hinge loss and regularization of 0.5 and use grid search to identify the best combination of parameters and features. we also compare two norwegian bert models: norbert foot_2 @xcite and nb-bert foot_3 (kummervold et al., 2021), which use the same architecture as bert base cased @xcite . norbert uses a 28,600 entry norwegian-specific sentence piece vocabulary and was jointly trained on 200m sentences in bokmål and nynorsk, while nb-bert uses the vocabulary from multilingual bert and is trained on 18 billion tokens from a variety of sources foot_4 , including historical texts, which presumably contain more examples of written dialect. we use the huggingface transformers implementation and feed the final '[cls]' embedding to a linear layer, followed by a softmax for classification. the only hyperparameter we optimize is the number of training epochs. we use weight decay on all parameters except for the bias and layer norms and set the learning rate for adamw @xcite to 1e-5 and set all other hyperparameters to default settings. we train the model for 20 epochs, and keep the model that achieves the best macro f 1 on the dev set. ## error analysis figure 1 shows a confusion matrix of nb-bert's predictions on the test data. the main three categories (bokmål, nynorsk, and dialect) are generally well predicted, while mixed is currently the hardest category to predict. this is expected, as the mixed class comprises all of the three other forms. the model has a tendency to predict nynorsk or mixed for dialect and struggles with mixed, predicting either bokmål or dialect. the same observations apply to norbert, mnb, and svm classifiers. given that our main interest lies in the ability to predict future dialect tweets, we compute precision, recall, and f 1 on only this label. the nb-bert model achieves 0.82, 0.91, and 0.86, respectively while norbert follows with 0.84, 0.77, and 0.81. the svm model achieves 0.80, 0.69, and 0.74 respectively, while mnb obtains slightly less scores with respectively 0.77, 0.66, and 0.71. this suggests that future experiments should consider using nb-bert. ## conclusion and future work in this paper we have described our first annotation effort to create a corpus of dialectal variation in written norwegian. in the future, we plan to use our trained models to expand the corpus in a semi-supervised fashion by refining our searches for tweets with dialectal traits in order to have a larger corpus of dialectal tweets, effectively pursuing a high-precision low-recall path. in parallel, we will begin to download large numbers of tweets and use our trained models to automatically annotate these (low-precision, high-recall). at the same time we plan to perform continuous manual evaluations of small amounts of the data in order to identify a larger variety of dialectal tweets, which we will incorporate into the training data for future models. second, we would like to annotate these dialectal tweets with their specific dialect. to avoid collecting too many tweets from overrepresented dialects, we will first annotate the current dialectal tweets with their dialect, and perform a balanced search to find a similar number of tweets for each dialect. finally, we would like to incorporate texts from different sources which contain rich dialectal variation, as e.g. books, music, poetry.