Murreviikko - A Dialectologically Annotated and Normalized Dataset of F innish Tweets (2023)

Abstract: This paper presents Murreviikko, a dataset of dialectal Finnish tweets which have been dialectologically annotated and manually normalized to a standard form. The dataset can be used as a test set for dialect identification and dialect-to-standard normalization, for instance. We evaluate the dataset on the normalization task, comparing an existing normalization model built on a spoken dialect corpus and three newly trained models with different architectures. We find that there are significant differences in normalization difficulty between the dialects, and that a character-level statistical machine translation model performs best on the Murreviikko tweet dataset.

## introduction dialectal variation is typical of user-generated content on social media, alongside other types of variation such as misspellings and emojis. such language can be challenging for natural language processing tools that are trained on standard language. we present a dataset of dialectal finnish tweets which have been manually annotated by dialect and normalized to standard finnish spelling. the dataset can be used as a test set for further work in, for instance, dialect identification or dialect-tostandard normalization. we further experiment with the latter, testing four different methods to normalize the tweets automatically: the publicly available rnn-based murre normalizer @xcite , a statistical machine translation system, a transformer-based neural machine translation system, and a normalizer based on the pre-trained byt5 model. to give an example of the task, the original dialectal text oonko mää nähäny should be replaced with the standard form olenko minä nähnyt ('have i seen'). the main contributions of the paper are: ## murreviikko murreviikko ('dialect week') is a twitter campaign initiated at the university of eastern finland which aims to promote the use of dialects in finland on social media. the campaign has run for three years @xcite @xcite and lasts for one week in october. ## results and discussion the chrf2 scores for the complete datasets are presented in 4 are not directly comparable. to see how our models' performance relates to theirs, we present the word error rates of the models in to further analyze the difficulty of the tweet normalization task, we scrutinize the normalization performance on the different dialect groups to see if some dialects are inherently harder to normalize, or if some models fail on some dialects. the chrf2 scores broken down by dialect are presented in the baselines reflect that the south-eastern (lai 67.3) and especially south-western dialects (lai 59.3) are further from standard finnish than the other dialects. both dialects include for instance eli-model sw hä so no fn he sa se na lai 59.3 71.7 73.1 74.9 75.6 73.9 74.2 67.3 83.7 regarding model performance, the helsinki slang (he) offers an interesting challenge. all models except byt5 perform worse than the baseline. this is somewhat to be expected, as the training data does not include the slang. byt5 on the other hand has been trained on web data by common crawl @xcite , which could include text written in the helsinki slang. it could also be that the swedish training data is helpful for the normalization task, since helsinki slang is characterized by swedish loanwords. the difficulty of the south-western dialects is reflected in the model scores, with all models achieving f-scores below 80. given this is the second largest dialect group in the dataset, it also affects the overall performance quite significantly. the character error rates for the complete datasets and dialects separately are presented in ## conclusions in this paper, we present a dataset of dialectal finnish tweets which have been manually annotated by dialect and normalized to a standard form. the dataset will be made accessible to the scientific community for further testing and fine-tuning of models in the fields of dialect-to-standard normalization and dialect identification, for instance. we furthermore evaluate four automatic normalization methods, which have been trained with transcribed spoken dialect data. three of the models have been purpose-built for this paper, while a fourth model has been made publicly available @xcite . character-level statistical machine translation provides the best normalization quality of the evaluated models on the murreviikko-dataset, with the pre-trained and fine-tuned byt5 model achieving very similar scores. meanwhile, the byt5 and a transformer-based neural model perform best on the test set of the dialect transcriptions (skn). the nmt model fails to transfer the performance to the tweets, however, consistently undernormaliz-ing and barely outperforming the baseline. the rnn-based murre normalizer struggles with the special characters typical of social media, while providing a reasonable performance on dialectal morphological features. dialect-wise, the south-western dialects provide the lowest baseline and worst scores for the models. in the context of this work, it is thus the hardest to normalize from the traditional finnish dialects. helsinki slang, traditionally not seen as one of the dialects, is also difficult for the models but this is mostly due to a lack of training data.