S wiss G erman Speech Translation and the Curse of Multidialectality (2025)

Abstract: In many languages, non-standardized varieties make the development of NLP models challenging. This paper explores various fine-tuning techniques and data setups for training Swiss German to Standard German speech-to-text translation models. While fine-tuning on all available Swiss German data yields the best results, ASR pre-training lowers performance by 1.48 BLEU points, and jointly training on Swiss and Standard German data reduces it by 2.29 BLEU. Our dialect transfer experiments suggest that an equivalent of the Curse of Multilinguality (Conneau et al., 2020) exists in dialectal speech processing, as training on multiple dialects jointly tends to decrease single-dialect performance. However, introducing small amounts of dialectal variability can improve the performance for low-resource dialects.

## introduction swiss german (schweizerdeutsch) is considered one of the most distinct and lively varieties of german with unique features on the phonological, morphological, syntactic and lexical levels 1 . it is a continuum of mostly high alemannic german dialects in switzerland, spoken by more than 5 million people. swiss german is used extensively in everyday situations, including spoken communication, text messaging, local and national tv programs, and even regional parliaments. standard german (hochdeutsch) is used for formal and institutionalized forms of communication @xcite . this coexistence of two varieties with clearly separated use cases in a single speaker group has been described as diglossia by several researchers @xcite @xcite ). 1 for a complete list of swiss german particularities, we refer the reader to @xcite and @xcite . swiss german dialects can vary significantly within switzerland, sometimes even leading to difficulties in understanding between swiss german speakers from distant regions @xcite . due to particularities on all linguistic levels, swiss dialects are hard to understand for many german speakers outside of switzerland @xcite and german learners who are primarily familiar with standard german @xcite . this makes the need for systems that can translate from swiss german speech to standard german text apparent. it could facilitate the integration of non-swiss-german speakers into swiss society by enabling them to understand local tv programs, radio shows, dialectal voice messages, and conversations between their co-workers. furthermore, dialectal speech translation can help preserve dialectal varieties and make language technologies more accessible to dialect speakers, contributing to the development of fair and equitable technologies @xcite . in a study by @xcite , 61% of respondents were in favor of systems that can translate dialect speech to standard german text. this highlights the demand for dialectal translation systems beyond academic interests. in the case of swiss german, automatic speech recognition (asr) and speech translation (st) are closely related. as swiss german does not have any standardized written form and all of its speakers understand standard german @xcite , it seems natural to prioritize swiss german speech to standard german text st instead of swiss german speech to swiss german text asr. although there are works about the latter @xcite , st is the subject of most research @xcite @xcite and was one of the shared tasks at the swiss text analytics conference foot_0 in 2021 @xcite and 2022 @xcite . although the area is being actively researched, swisstext 2022 @xcite has demonstrated that the problem is far from being solved. none of the participating teams were able to outperform the baseline model, a simple transformer fine-tuned on three datasets. later works achieved improvements over this baseline by using more data and experimenting with fine-tuning pre-trained models @xcite . however, they did not explore further pre-training, nor did they utilize all the available data for swiss german, or employ standard german data. @xcite showed that one of the main challenges is that swiss german st needs to handle a considerable amount of dialectal variability. they found that some dialects benefit from positive transfer from related dialects, whereas others negatively influence overall performance. it remains unclear, however, how many dialects can be used together to improve performance and when performance starts to degrade. here, we expect a breaking point as observed for the curse of multilinguality @xcite even for the closely related swiss dialects. furthermore, we don't know how small amounts of dialectal variability affect performance. we aim to close these research gaps by: researchers have proposed various techniques to mitigate performance drops due to multidialectality, with a primary focus on automatic speech recognition (asr). using pre-trained models has been found to outperform monolingual training from scratch @xcite . @xcite suggest dialect-aware asr modeling by simultaneously performing dialect identification and asr for japanese dialects, @xcite , @xcite , and yadavalli et al. (2022) apply similar multi-task training approaches to chinese, english, and telugu. using the standard and dialectal varieties together during training has been found to increase performance for tunisian arabic @xcite , for multiple other arabic dialects @xcite , and for thai when combined with curriculum learning 3 suwanbandit et al. (2023). ## swiss german st for german, research in dialectal speech processing is scarce. wepner (2021) calls for adapting asr systems to austrian german as they observe a performance discrepancy between german standard german and austrian standard german. similarly, @xcite find an increase of 24.8% in wer when evaluating a german asr system on dialectal utterances, and @xcite see the need to include dialectal varieties in german asr datasets. @xcite find that the multidialectal nature of swiss german, briefly described in the introduction, is one of the main challenges for swiss german st. they observe positive and negative transfer between dialects, mainly depending on their overall similarity as determined by @xcite . swiss german st is actively researched, and many datasets have been released in the past years 4 . abbr. dataset total h train h cantons t stt stt4sg-350 (plüss et al., 2023a) 343 239 17 stg sds sds-200 (plüss et al., 2022) 200 50 21 stg spc swiss parliaments corpus (plüss et al., 2020) 293 217 n/s stg sdial swissdial (dogan-schönberger et al., 2021) 36 36 8 stg grzh gemeinderat zürich corpus (plüss et al., 2021) 1208 1208 n/s -am archimob (samardzic et al., 2016) 80 0 14 swg -total data with standard german labels 872 542 -stg early work on swiss german to standard german st has focused on single dialects and pipeline systems @xcite , as st data was scarce. however, @xcite emphasize that the lack of a standard orthography and limited resources make it difficult to train cascade systems, making end-to-end architectures dominate the swiss german st area @xcite @xcite . current state-of-the-art models for swiss german st mostly follow the pre-train and fine-tune paradigm. @xcite fine-tune an xls-r 1b model on the stt dataset and achieve state-ofthe-art performance on the sds, stt, and swiss-text2021 test sets (69.6 bleu, 74.7 bleu, and 66 bleu, respectively). @xcite find that whisper exhibits strong zero-shot capabilities for swiss german, outperforming the previously mentioned model on the spc test set. @xcite trained three small models on the stt data, with xls-r 0.3b outperforming whisper s and a transformer model trained from scratch. these ian, swabian, and alsatian dialects. findings make it difficult to determine which architecture is the most suitable for swiss german st. furthermore, recent pre-trained multilingual models, such as seamlessm4t @xcite and audiopalm @xcite , have not yet been evaluated for this task. ## data and models in this section, we detail the models and datasets used for our speech-to-text translation experiments for swiss german. the methodology used for the experiments will be described in section 5 and 6. ## fine-tuning experiments to improve the state-of-the-art of swiss german st and investigate whether using data from a closely related language (standard german) is beneficial for st performance, we conduct a series of experiments. experiments 1-4 focus on different finetuning strategies and data setups, while experiment 5 involves continued pre-training of xls-r. all experiments aim to improve overall swiss german translation performance and train robust models that perform well across different data domains. ## dialect transfer experiments in these experiments, we vary the number and diversity of dialects in the training data to study the effect of dialectal variability on performance and determine if there is an equivalent to the curse of multilinguality @xcite for dialects. ## conclusion and future work with respect to the research gaps identified in the introduction, the main findings of this paper are the following: