Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal B avarian Case Study (2025)

Abstract: Reliable slot and intent detection (SID) is crucial in natural language understanding for applications like digital assistants. Encoder-only transformer models fine-tuned on high-resource languages generally perform well on SID. However, they struggle with dialectal data, where no standardized form exists and training data is scarce and costly to produce. We explore zero-shot transfer learning for SID, focusing on multiple Bavarian dialects, for which we release a new dataset for the Munich dialect. We evaluate models trained on auxiliary tasks in Bavarian, and compare joint multi-task learning with intermediate-task training. We also compare three types of auxiliary tasks: token-level syntactic tasks, named entity recognition (NER), and language modelling. We find that the included auxiliary tasks have a more positive effect on slot filling than intent classification (with NER having the most positive effect), and that intermediate-task training yields more consistent performance gains. Our best-performing approach improves intent classification performance on Bavarian dialects by 5.1 and slot filling F1 by 8.4 percentage points.

## introduction most research on natural language processing (nlp) for digital assistants has focused on standardized languages, despite the large degree of dialectal variation exhibited by many languages and the positive attitude towards dialectal versions of such technologies expressed by some speaker communities @xcite . a core task of natural language understanding (nlu) is to detect the intent of an input to a digital assistant (e.g., the instruction "delete all alarms" belongs to the cancel alarm class) and to tag it for specific slots (e.g., "all" should be tagged as the reference associated with the intent). however, classifying dialectal inputs is still challenging we finetune pre-trained language models (plms) on english sid data (grey ○) and evaluate them on bavarian (red ○). we compare multiple setups: a) no auxiliary tasks, b) multi-task learning by jointly training on english sid data and bavarian auxiliary tasks ("aux"), c) intermediate-task training on bavarian, then finetuning on english sid data. as contemporary models are less proficient due to the scarcity of low-resource and especially dialectal training data @xcite . to overcome this issue, transferring task knowledge cross-lingually from high-resource language data to low-resource varieties is a strategy widely used in nlu @xcite @xcite . while many approaches have focused on cross-lingual transfer via embedding transmission and machine translation, van der goot et al. (2021a) use non-english auxiliary task data for zero-shot transfer to other languages. inspired by this setup and by intermediate-task training procedures @xcite , we use auxiliary tasks to analyze and improve zeroshot transfer learning for slot and intent detection (sid) for bavarian dialects (figure 1 ). to account for intra-dialectal variation, we evaluate on two previously released bavarian datasets and introduce a third test set. for the auxiliary tasks, we use three recent bavarian datasets for syntactic annotations, named entity recognition (ner), and masked language modelling (mlm). we make the following contributions: we share our code publicly. 2 2 related work slot and intent detection for dialects and nonstandard varieties research on sid for lowresource languages, including non-standard and dialectal varieties, has started receiving more attention. this trend starts with van der @xcite , who introduce a multilingual sid dataset, xsid, containing south tyrolean, a bavarian dialect (more details in §4.1). xsid has since been extended with dialectal data from upper bavaria @xcite , data in bernese swiss german and neapolitan @xcite , and eight norwegian dialects @xcite . similarly to our study, van der goot et al. (2021a) experiment with multi-task learning, although they only have standard german auxiliary data at their disposal for the south tyrolean test data. other approaches focus on tokenization issues or data augmentation. @xcite tackle tokenization issues caused by spelling differences by injecting character-level noise into standardlanguage training data, which improves the performance on the dialectal test sets. @xcite find that encoding text with visual representations (rather than ones based on subword tokens) improves transfer from standard german to german dialects for intent classification. @xcite fine-tune a masked language model on dialectal data to generate synthetic training data for german and arabic dialects. @xcite organized a shared task on intent detection in four arabic dialects, where the top systems all involve model ensembling and translating the training data into the test dialects @xcite @xcite . in the context of spoken intent classification, other work focuses on variation in spoken italian @xcite and english @xcite @xcite . multi-task learning (mtl) joint mtl learning involves jointly training a model on several tasks. @xcite provides a general overview. martínez alonso and plank (2017) find that tasks with non-skewed label distributions lend themselves best as auxiliary tasks for sequence tagging. @xcite show that auxiliary tasks which are more similar to the target tasks result in better target performance. regarding mtl for sid, wang et al. ( 2021 ) train a transformer model on dependency parsing, pos tagging, and sid, with different layers attending to the different tasks. they find that the syntactic tasks improve sid performance (especially when both are included), and that jointly producing slot and intent labels is also beneficial. @xcite use english training data for sid but additionally exploit non-english auxiliary task data, hypothesizing that this helps their models to learn additional linguistic properties of the target language. they find syntactic tasks to be useful for slot filling for one pre-trained language model but not another, and harmful for intent detection. similarly, they find masked language modelling (mlm) to be of use for slot filling but not intent classification. machine translation as auxiliary task yielded worse performance. ## background: bavarian dialects bavarian dialects differ from standard german in phonetics, phonology, word choice, and morphosyntax @xcite . there is no established orthography or standard variety of bavarian. the bavarian dialects belong to the upper german dialect group and are split into three major subgroups (northern, central, and southern bavarian; figure 2 ), mostly based on sound differences @xcite . there is also phonetic/ phonological and lexical variation within these groups @xcite . the pronunciation differences are also reflected in the spelling choices made in the different training and evaluation datasets in our study, although the spellings also reflect idiosyncratic preferences. we compare the bavarian sid test sets in §6.4. some of the morphosyntactic differences between bavarian and standard german (cf. @xcite are relevant for sid, and recent work @xcite has shown that slot filling performance in german is negatively affected by dialectal syntactic structures. person names are typically preceded by definite articles, and the given name generally follows the family name @xcite -this has been analyzed in the context of ner @xcite and might also be relevant for slot filling. furthermore, many nlu queries contain infinitive constructions of the form "remind me to [do x]". such cases are often expressed with a nominalized infinitive construction @xcite ; see, e.g., additionally, as in many other german dialects @xcite , temporal expressions (relevant for datetime slots) can be expressed in ways that are not grammatical in standard german, e.g., fia fünfe heid auf nacht "for 5pm tonight" (lit. "for five today at night") or um 3 nammiddog "at 3pm" (lit. "at 3 afternoon"). ## methodology we fine-tune pre-trained language models (plms) on xsid's english training data using machamp 0.4.2 (commit 9f5a6ce; van der @xcite with the same hyperparameters as van der goot et al. (2021a) did for their sid experiments. we evaluate slot predictions with strict slot f1, intent predictions with accuracy, and also calculate the proportion of sentences with fully correct predictions. we treat sid itself as a multi-task setup as we jointly predict the slots and intent labels, and treat slot detection as a basic sequence labelling task with a final softmax layer. we use the following task types for machamp @xcite : seq (slot filling, ner, pos tagging), classification (intent classification), mlm (mlm), and dependency (dependency parsing). the loss for each task is weighted equally. we use machamp's default loss functions (cross-entropy loss for all tasks except dependency parsing, which uses negative log likelihood). we provide mean scores across three runs for each experiment. we compare three types of experimental setups (figure 1 ): baseline we compare four commonly used plms, which we finetune on sid data without auxiliary tasks: the monolingual german gbert @xcite , and the multilingual models mbert @xcite , xlm-r (conneau et al., 2020), and mdeberta v.3 @xcite . notably, mbert's pretraining data also includes the bavarian wikipedia, which contains articles in all three of our test dialects. xlm-r and mde-berta were pre-trained on the cc-100 dataset @xcite , which does not contain bavarian data. gbert's pretraining data is in standard german. to limit computation costs, we use the base-sized versions. 4 in the remaining setups, we only use mdeberta because of its strong performance as a baseline plm ( §6.1). multi-task learning we train the model to jointly predict labels for sid and at least one auxiliary task. we use × to denote these setups, e.g., ner×sid refers to training a model to simultaneously predict named entity labels, slots and intents. intermediate-task training we first train the model to predict labels for an auxiliary task, remove the task-specific head, optionally repeat this for a second auxiliary task, and then finally train the model to predict sid labels. we @xmath0, e.g. @xmath1, then on sid data. as a special case, we train some models first jointly on auxiliary tasks and then afterwards on sid (e.g., mlm×@xmath2. we apply each auxiliary task dataset to both finetuning setups. for the settings with multiple auxiliary tasks, we select combinations that appear promising based on the results already obtained. we were not able to examine all possible combinations due to computational restraints. ## results and analysis we first present the results of the baseline models ( §6.1), and then discuss the impact of finetuning the model on auxiliary bavarian nlp tasks ( §6.2). we next compare performances across table the bavarian dialects as well as an additional upper german dialect (bernese swiss german) and the standard languages german and english ( §6.3). we additionally discuss differences between the bavarian translations ( §6.4), and lastly analyze the results on other bavarian sid datasets ( §6.5). ## conclusion in all of our cross-lingual sid experiments, the performance patterns are similar across dialects, but the actual scores differ. to allow future research on this kind of variation, we release a new evaluation dataset (de-muc). in our experiments, intermediatetask training tends to produce better results than joint multi-task learning. additionally, our bavarian auxiliary tasks (pos tagging and dependency parsing, ner, mlm) were more beneficial for slot filling than intent classification, with ner being the overall most helpful auxiliary task.