C or C o D ial - Machine translation techniques for corpus-based computational dialectology (2023)

Abstract: This paper presents CorCoDial, a research project funded by the Academy of Finland aiming to leverage machine translation technology for corpus-based computational dialectology. In this paper, we briefly present intermediate results of our project-related research.

## introduction dialectology is concerned with the study of language variation across space. over the last decades, dialectologists have collected large datasets, which typically consist of transcribed interviews with informants. unfortunately, these interviews cannot easily be compared with each other as they differ considerably in length and content. if informant a does not use word x, this does not necessarily mean that the word does not exist in a's dialect. it may just be that a chose to talk about topics that did not require the use of word x. the corcodial (corpus-based computational dialectology) project aims to introduce comparability in dialect corpora with the help of machine translation techniques. corcodial is funded by the academy of finland during the period 2021-2025. the core of the project focuses on the dialectto-standard normalization process, which is a sequence-to-sequence task that maps the phonetic transcriptions to the standardized spellings. we are not only interested in the results of the normalization process, but also in the emerging representations of dialects and speakers that the (statistical or neural) normalization models learn. these representations allow us to provide new visualisations of dialect landscapes and to confirm or challenge traditional dialect classifications. traditional dialect corpora are costly to produce: informants need to be found and inter-viewed, and the recorded interviews need to be transcribed and annotated. to circumvent this data bottleneck, researchers have increasingly turned to user-generated content (ugc), i.e., to texts published by laypeople on social media. we also investigate to what extent normalization methods trained on "clean" data transcribed by dialectologists generalize to noisier ugc datasets. the main goals of the corcodial project are: 1. to improve the automatic normalization of dialect texts by using state-of-the-art machine translation methods, 2. to extract, visualize, compare and interpret the dialectal patterns emerging from the normalization models, and 3. to contrast the patterns found in traditional dialectological corpora with those found in user-generated content. in the following sections, we present some results of our ongoing research. ## benchmarking dialect-to-standard normalization systems in contrast to historical text normalization (bollmann, 2019; @xcite and ugc standardization, there have not been any multilingual evaluations of dialect-to-standard normalization systems. in order to establish dialect normalization as a distinct task, we compiled a multilingual benchmark dataset from existing sources, covering finnish, norwegian, swiss german and slovene. we evaluate different sequence-to-sequence models that have been previously employed for normalization tasks: 1 statistical machine translation with character-level segmentation; neural machine translation with rnn and transformer architectures, character-level and bpe segmentation, and full-sentence and word-trigram windows; and the pre-trained multilingual byt5 model using byte-level segmentation. our results show that the transformer is the most successful model architecture on all four datasets. this is somewhat surprising since recent related work @xcite @xcite found smt and rnn-nmt to be competitive. using word trigram windows instead of full sentences, as in @xcite , is also effective in our setup, although the gap towards full-sentence models is considerably lower than in their work. finally, the pre-trained byt5 model only outperforms vanilla transformers on the norwegian dataset. ## analyzing speaker representations in multi-dialectal nmt language labels are often used in multilingual neural language modeling and machine translation to inform the model of the language(s) of each sample. as a result of the training process, the models learn embeddings of these language labels, which in turn reflect the relationships between the languages @xcite . following @xcite , we apply this idea to the finnish and norwegian parts of the normalization dataset introduced in the previous section. we use distinct labels for each speaker in the corpus and analyze their representations obtained by the transformer-based normalization models. we find that (1) the speaker label embeddings of two speakers coming from the same village are very similar, and that (2) the embeddings of all speaker labels taken together reflect the traditional dialect classifications precisely. detailed results of this analysis are given in @xcite . in order to extend our dialectological research to more modern and realistic types of data, we collected and annotated a dataset of dialectal finnish tweets. we take advantage of murreviikko ('dialect week'), a twitter campaign initiated at the university of eastern finland, which promotes the use of dialects on finnish social media. the campaign lasts for a week in october and has run for three years (2020-2022). we collected tweets containing the keyword murreviikko or #murreviikko via the twitter api from all three years. this collection resulted in a total of 465 tweets, 344 of which were written in a dialect of finnish. the tweets were manually annotated by a dialectologist with the dialect region and normalized to standard finnish on sentence level. in contrast to the "clean" finnish dialect dataset used in our benchmark (section 2), the murreviikko data is much noisier. 2 in terms of normalization performance, the smt model has been found to perform best, followed by the pre-trained byt5 model. these two approaches turned out to be much more robust to noise than the vanilla transformers. the corpus collection process, the normalization results and the modalities of access are described in detail in @xcite . 3