Dialect Representation Learning with Neural Dialect-to-Standard Normalization (2023)

Abstract: Language label tokens are often used in multilingual neural language modeling and sequence-to-sequence learning to enhance the performance of such models. An additional product of the technique is that the models learn representations of the language tokens, which in turn reflect the relationships between the languages. In this paper, we study the learned representations of dialects produced by neural dialect-to-standard normalization models. We use two large datasets of typologically different languages, namely Finnish and Norwegian, and evaluate the learned representations against traditional dialect divisions of both languages. We find that the inferred dialect embeddings correlate well with the traditional dialects. The methodology could be further used in noisier settings to find new insights into language variation.

## introduction starting with @xcite , multilingual neural models have become increasingly popular for both language modeling and sequence-tosequence learning tasks. the most common type of multilingual model makes use of language labels that are prepended to the training and test instances to inform the model about the language being processed. the embeddings of the language models can then be analyzed to find emerging properties of the relationships between the languages @xcite . in this paper, we apply the same idea to a smaller granularity of linguistic variation, namely dialectal variation within a language area, and we use dialectto-standard normalization as the modeling task. focusing on two typologically different languages, we experiment with large datasets of finnish and norwegian dialects. we study the inferred dialect embeddings with different dimensionality reduction algorithms to see whether the neural normalization models learn dialectal differences. we find that the learned representations correlate well with the traditional dialect classifications. ## conclusions in this paper, we apply neural dialect-to-standard normalization models to two typologically different languages and use the learned speaker representations to study the dialect continuum and division of the languages. we use large datasets of norwegian and finnish dialects, which have been manually transcribed and manually or semi-automatically normalized to a standard form. we add speaker labels to each dialect utterance (source) and normalize to the standard language, using byte-pair encoded data. the model learns representations of the speakers based on the speaker labels added to the dialect utterances. the learned representations are further studied with principal component analysis, agglomerative clustering with ward linkage, and k-means clustering. the results are evaluated against gold standard divisions of the dialects using v-measure and adjusted rand index as metrics. agglomerative clustering with ward linkage outperforms k-means clustering for both languages on v-measure. we find that the learned representations of the speakers correspond well to traditional dialect divisions. we also show that some dialect areas, such as the häme dialect in finnish are not as homogenic as could be assumed by the traditional division. the methodology could be further used with noisier data from social media for instance, which could reveal new insights into areal variation.