Help from the Neighbors: E stonian Dialect Normalization Using a F innish Dialect Generator (2022)

Abstract: While standard Estonian is not a low-resourced language, the different dialects of the language are under-resourced from the point of view of NLP, given that there are no vast hand normalized resources available for training a machine learning model to normalize dialectal Estonian to standard Estonian. In this paper, we crawl a small corpus of parallel dialectal Estonian - standard Estonian sentences. In addition, we take a savvy approach of generating more synthetic training data for the normalization task by using an existing dialect generator model built for Finnish to “dialectalize” standard Estonian sentences from the Universal Dependencies tree banks. Our BERT based normalization model achieves a word error rate that is 26.49 points lower when using both the synthetic data and Estonian data in comparison to training the model with only the available Estonian data. Our results suggest that synthetic data generated by a model trained on a more resourced related language can indeed boost the results for a less resourced language.

## introduction estonian itself can hardly be characterized as lowresourced due to a variety of nlp tools @xcite @xcite and corpora @xcite @xcite available for the language. what still remains a difficult and severely under-resourced task to tackle is nonstandard dialectal language. estonian has a rich morphology which means that an individual word can have several different inflectional forms. in terms of dialects, this means that all of the different inflectional forms of a given word can be slightly different in different dialects of the language. this poses a challenge for nlp methods that are mostly trained on standard estonian. while the written standard is something people follow when they write official text such as pub-lished books or newspapers, people tend to communicate using dialect in more informal settings such as when sending messages or emails with friends and family or when engaging in discussion on online forums. this type of an every day language use is beyond the reach of current nlp methods for estonian. for other languages such as finnish @xcite , swedish @xcite and german @xcite , dialect normalization has been seen as good way of dealing with the issue of non-standard language. if a model can normalize dialectal text to a standard norm, then all normative language nlp models can be applied on that data. normalization has been shown to improve results in a variety of tasks such as parsing (van der goot et al., 2020) and neologism retrieval @xcite . unfortunately, estonian does not have vast dialectal resources available with aligned normalizations with dialectal sentences. for this reason, we establish a new methodology for producing synthetic dialectal estonian -standard estonian sentence pairs using a finnish dialect generation model. the data and the models presented in this paper have been released openly on zenodo foot_0 . estonian dialects are traditionally divided into northern and southern groups, that differ on phonological, morphological as well as on lexical levels. according to the general estonian dialect classification @xcite , there are three main dialect groups. (1) the north estonian dialect group consists of the eastern, insular, central, and western dialects. (2) the northeastern coastal dialect group consists of the northeastern and the coastal dialects. (3) the south estonian group consists of the mulgi, tartu, and võru dialects. in recent decades, the question of seto has been debated. the distinction between seto and võru has been justified for instance on a syntactic level @xcite . the dominant contact languages for estonian dialects are swedish, russian, latvian, and votic. finnish, ingrian and livonian have influenced somewhat less @xcite ). ## related work there have been several different approaches to text normalization in the past @xcite . in this section, we will give a quick overview of the common approaches. dialectal text normalization has been tackled by using normalization rules and heuristics @xcite @xcite . later on, algorithmic approaches have been used for the task @xcite @xcite . very frequently, normalization is modeled as a character-level machine translation task. there are several research papers that use a statistical machine translation approach with a character level ngram language model of varying lengths @xcite @xcite . more recently, neural machine translation has been used on a character level for the normalization task @xcite @xcite . the approaches consist typically of a bi-directional lstm model and an attention mechanism. this approach has also been used with word2vec to extract and train an ocr post-correction model in an unsupervised way @xcite . with the emergence of general purpose language models, many recent papers present work on using such models for text normalization. bert @xcite , bart @xcite and roberta @xcite , for instance, have all been use lately to solve the task. ## dialect data since there is no dialectal corpus with standard normalizations available for estonian, we have to crawl one relying on the accessible resources. the institute of estonian language has released some dialectal dictionaries online foot_1 . some of these contain example sentences in one of the dialects and their normalization in standard estonian. in particular, we found that the dialectal dictionaries for mulgi foot_2 , kihnu foot_3 and hiiu foot_4 dialects had such aligned dialectal-standard estonian sentence pairs. we proceeded to crawl the aforementioned dictionaries. the dictionaries do not have an index of lemmas or any other means of browsing them apart from search queries. for this reason, we use the full text query the online dictionaries have to find occurrences of a given word in anywhere within the dictionary entries. we do this query for the 10,000 most frequent words foot_5 recorded in the eesti kirjakeele sagedussõnastik @xcite which is based on a relatively large 1 million word corpus. this crawling approach leads to the same texts being crawled multiple times, and for this reason, we remove all duplicates. some of the dialectal example sentences have additional annotation such as stress marked on top of the vowels. we clean the data of any additional marking and punctuations so that we are left with characters that are part of the estonian alphabets. furthermore, we ensure that the dialectal sentence and its normalization have an equal number of words. this step is needed because sometimes the example sentences were not normalized or were not fully normalized. this way we can clear all wrongly aligned sentences from the data. this resulted in 14510 aligned dialectal-normative estonian sentences. in we split the corpus randomly to 70% training, 15% validation and 15% testing. this split is used for all the models we train that include estonian data in their training. all models are evaluated with this test split. dialectal normalized translation na joove kõrdamisi ütest laasist nad joovad kordamööda ühest klaasist they take turns drinking from one glass siis oli tiädmätä jäen ning oksõndan siis oli teadvuse kaotanud ja oksendanud then he had lost consciousness and vomited ärä tettä alatude inemistege tegemist ära tee alatute inimestega tegemist don't deal with naughty people ## dialect normalization we train bert-based @xcite models to do estonian normalization using transformers python library @xcite . we model the task as a sequence to sequence task, where the model is trained to predict a normalized version of a sentence given a dialectal sentence. the model consists of a bert based encoder and decoder models similarly to the architecture proposed in @xcite . we build our models on estbert foot_6 (tanvir et al., 2021) which is a bert model trained solely on estonian data using the estonian national corpus. we train three models: one with estonian only data, one only with synthetic data and one with both types of data. we train the models for 3 epochs. ## results and evaluation in this section, we present the results of our models using wers. word error rate foot_8 is a commonly used metric to assess the quality of normalization models as it shows how far away the normalization predicted by a computational model is from the ground truth in terms of substitutions, insertions and deletions. we also calculate a token level accuracy which shows how many times a token was correctly normalized in the exact position it appeared in the sentence. the results can be seen in ## conclusions we have shown that despite estonian not having enough data on its own to train a dialect normalization model, using a finnish dialect generator model with some orthographic conversion rules to produce synthetic data can boost the results. although the results were promising, the best wer is still relatively high. this is partially due to the normalization strategy used in the original data. nevertheless we believe that experimenting more with synthetic data in the future can help us push the wer lower.