Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation (2023)

Abstract: Text normalization methods have been commonly applied to historical language or user-generated content, but less often to dialectal transcriptions. In this paper, we introduce dialect-to-standard normalization – i.e., mapping phonetic transcriptions from different dialects to the orthographic norm of the standard variety – as a distinct sentence-level character transduction task and provide a large-scale analysis of dialect-to-standard normalization methods. To this end, we compile a multilingual dataset covering four languages: Finnish, Norwegian, Swiss German and Slovene. For the two biggest corpora, we provide three different data splits corresponding to different use cases for automatic normalization. We evaluate the most successful sequence-to-sequence model architectures proposed for text normalization tasks using different tokenization approaches and context sizes. We find that a character-level Transformer trained on sliding windows of three words works best for Finnish, Swiss German and Slovene, whereas the pre-trained byT5 model using full sentences obtains the best results for Norwegian. Finally, we perform an error analysis to evaluate the effect of different data splits on model performance.

## introduction text normalization refers to a range of tasks that consist in replacing non-standard spellings with their standard equivalents. this procedure is beneficial for many downstream nlp tasks, since it increases data homogeneity and thus reduces the impact of unknown word forms. establishing identity between different variants of the same form is also crucial for information retrieval tasks, and in particular for building efficient corpus querying systems. furthermore, it can facilitate building applications for a wider audience, such as spelling and grammar checkers. important progress has been made on normalization of historical texts (e.g., @xcite and user-generated content (ugc) in social media (e.g. @xcite . however, the existing work on dialect normalization (e.g., @xcite @xcite remains fragmented: it typically focuses on a single language and uses different models, experimental setups and evaluation metrics, making direct comparisons difficult. in this paper, we aim to establish dialect-tostandard normalization as a distinct task alongside historical text normalization and ugc normalization. we make the following contributions: ## datasets we propose a multilingual dataset that covers finnish, norwegian, swiss german and slovene. the dataset is compiled from existing dialect corpora, which are presented in detail below. the languages originate from two language families (uralic and indo-european) and three branches (finnic, germanic, balto-slavic). all languages adopt the latin script. this enables the comparison of language structure, rather than differences in script. some quantitative information about the individual corpora is available in ## methods and tools we utilize both statistical and neural machine translation tools trained from scratch, as well as a pretrained multilingual model. we train (or fine-tune) the models for each of the four languages separately. our hyperparameter choices largely follow recent related work on text normalization. the main characteristics of the models are summarized below, and a detailed description of the hyperparameters is given in appendix a. the methods used in the experiments are: smt. our statistical machine translation method corresponds mostly to the one implemented in the csmtiser tool. 10 it uses the moses smt toolkit @xcite with a 10-gram kenlm language model trained on the training sets. 11 @xcite found eflomal @xcite to produce better character alignment than the more commonly used giza++, and we adopt this method. minimum error rate training (mert) is used for tuning the model weights, using wer (word error rate, which effectively becomes character error rate in a character-level model) as the objective. rnn-based nmt. this model uses a bidirectional lstm encoder and a unidirectional lstm decoder with two hidden layers each. the attention mechanism is reused for copy attention. tf-based nmt. this model has 6 transformer layers in the encoder and the decoder, with 8 heads each. we found in preliminary experiments that position representation clipping was beneficial to the results. all nmt models are trained with the opennmt-py toolkit @xcite . byt5. byt5 is a multilingual pre-trained model of the t5 family @xcite . these models we run all models on our base setup (entire sentence instances with character or byte tokenization). in addition, the smt, rnn and tf models are also trained on subword-segmented data. finally, the character-level rnn and tf models are trained on sliding windows. 13 ## results and discussion we evaluate the models presented in section 5 with the metrics described in section 4.3. we run the models on the three folds of each data split, and present the average scores and standard deviations for each metric. the word-level accuracies are presented in while csmt and rnn-based methods do not yield the best score for any dataset, they still perform reasonably well. for the nmt models, using the sliding window instead of sentences enhances the results for all corpora but ndc. regarding tokenization, the subwords improve the performance on the large corpora (skn and ndc) but worsen it on the small corpora (archimob and gos). for all datasets, the best results are obtained on the character (or byte) level. we expected rising levels of difficulty between data split 1 and 3, but neither the baselines nor the model outputs confirmed our expectations. the the character error rates presented in ## conclusions in this paper, we present the dialect-to-standard normalization task as a distinct task alongside historical text normalization and ugc normalization. we introduce a dialect normalization dataset containing four languages from three different language branches, and use it to evaluate various statistical, neural and pre-trained neural sequence-to-sequence models. in our base setup with models trained on entire sentences with character (or byte) segmentation, the pre-trained byt5 model performs best for all languages and data splits. moving from character segmentation to subword segmentation increases the accuracies for the large datasets (skn and ndc), but not enough to surpass byt5. in contrast, the sliding window approach outperforms byt5 on all languages except norwegian. the superior performance of byt5 on norwegian cannot be directly explained by the amount of training data, foot_15 but it is likely that the closely related languages swedish and danish enhance its performance. a further analysis on character error rate shows that the neural models sometimes offer very poor predictions, which are not visible when using accuracy as a metric. in this work, we have evaluated the most common and most popular model architectures, but it would be interesting to test specific model architectures for character transduction tasks, e.g. models that put some monotonicity constraint on the attention mechanism @xcite . we defer this to future work. another point to be investigated in future work is data efficiency. our training sets are relatively large in comparison with other character transduction tasks, and it would be useful to see how much the data requirements can be reduced without significantly affecting the normalization accuracy.