Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial G erman Varieties (2024)

Abstract: Mainstream cross-lingual task-oriented dialogue (ToD) systems leverage the transfer learning paradigm by training a joint model for intent recognition and slot-filling in English and applying it, zero-shot, to other languages. We address a gap in prior research, which often overlooked the transfer to lower-resource colloquial varieties due to limited test data. Inspired by prior work on English varieties, we craft and manually evaluate perturbation rules that transform German sentences into colloquial forms and use them to synthesize test sets in four ToD datasets. Our perturbation rules cover 18 distinct language phenomena, enabling us to explore the impact of each perturbation on slot and intent performance. Using these new datasets, we conduct an experimental evaluation across six different transformers. Here, we demonstrate that when applied to colloquial varieties, ToD systems maintain their intent recognition performance, losing 6% (4.62 percentage points) in accuracy on average. However, they exhibit a significant drop in slot detection, with a decrease of 31% (21 percentage points) in slot F _1 score. Our findings are further supported by a transfer experiment from Standard American English to synthetic Urban African American Vernacular English.

## introduction the usability of dialog systems heavily relies on the ability to handle user inputs in multiple languages. recent language models (lms) have become state-of-the-art tools to carry out the primary task-oriented dialogue (tod) problems, including intent recognition and slot filling. what is more, lms leverage multilingual pre-training to facilitate transfer across languages. to achieve this, the mainstream approach involves fine-tuning the lm on a pivot language, commonly english, and subse- spiele das von eddie vinson tue das vom eddie vinson spielen searchcreativework playmusic lied b-music_item b-artist b-contact lied do by + det.dat vinson eddie play the song by eddie the song vinson play b-artist i-artist b-music_item the top part displays the intact sentence with gold labels, the bottom part shows the prediction for the perturbed sentence. the perturbations tun _ imperative, article _ name, name _ order are applied. there are errors in predicting the intent and one of the two slots. while this language transferring approach has achieved impressive results for many language pairs, its effectiveness is limited when it comes to processing low-resource language varieties and dialects @xcite . these varieties are often underrepresented in the lm's pre-training data and may not align well with the characteristics of the chosen pivot language. our current understanding of how well modern lms handle dialects and the extent of disparity between standard languages and dialects remains limited. therefore, it is important to assess the performance gap in the first place, as highlighted by @xcite to identify key directions for further development. processing (non-standardized) dialects brings unique challenges: large volumes of writing such as newspapers or fiction are rarely produced, and access to conversational data in social media is limited. besides, dialects lack unified spelling rules @xcite and exhibit a high degree of variation over space and time @xcite . finally, dialects may additionally show a significant rate of code-mixing compared to standard languages @xcite . to tackle these challenges, recent studies have introduced techniques that mimic dialectal morphosyntactic variation through rule-based translation systems which perturb sentences into respective dialect variants in german @xcite and english @xcite . this approach is highly practical as it avoids the expense of annotating new data while still effectively stresstesting applications like question answering and machine translation @xcite . building on this, we choose tod as a task where we expect a high level of linguistic variation in reallife application settings @xcite . we experimentally evaluate how well tod systems handle dialectal data by simulating dialectal and colloquial variations in english and german to explore the following research questions (rqs). rq 1: how does the lm performance in intent recognition and slot filling change when applied to synthetic dialectal data in both english and german? rq 2: considering that each perturbation isolates a specific dialectal phenomenon, which perturbations have the most significant effect? rq 3: how do lms differ in terms of robustness to dialectal perturbations? figure 1 illustrates our approach. to address these rqs, we contribute in the following ways: (i) we define and implement a set of hand-crafted perturbation rules for translating from standard german to its spoken varieties ( §3.2). we release the code for the perturbation rules and the results of our experimental evaluation for further uptake: github.com/mainlp/dialect-tod-robustness. ## related work robustness of tod systems. the evaluation of task-oriented dialogue (tod) systems' robustness aims to investigate the generalization capabilities of lms and their ability to adapt to domain shifts, with a specific focus on english @xcite . the robustness of tod systems has been widely investigated using adversarial attacks, which involve manipulating the gradients and weights of lms to alter their predictions @xcite . nevertheless, white-box methods lack linguistic awareness, making them not easily interpretable @xcite . in contrast, recent black-box methods have emerged that aim to mimic language variation and real-life noise, including speech artifacts and typos, with the primary objective of crafting instances that deceive lms @xcite @xcite . a related line of research focuses on developing defenses against adversarial attacks and enhancing the robustness of tod systems by employing techniques such as data augmentations and incorporating regularization terms in the loss function @xcite . nlp for dialects and non-standard varieties. previous efforts in processing dialects and nonstandard varieties have primarily focused on differentiating between dialects and closely related languages. notably, the vardial initiative @xcite @xcite has conducted a series of evaluation campaigns aimed at dialect identification and discrimination between similar languages. additional research directions in the field include partof-speech (pos) tagging @xcite , syntactic parsing @xcite , low-resource intent identification and slot filling @xcite . moreover, machine translation techniques have been applied to re-write sentences from dialect to standard language @xcite @xcite . to overcome the limited availability of parallel training data, rule-based perturbations simulating dialectal morphosyntactic phenomena have been developed to generate synthetic parallel sentence pairs @xcite . the emergence of pre-trained lms has shifted the focus towards investigating disparities in representation and downstream performance between non-standard and standard languages. to this end, lm diagnostic tools encompass a wide range of techniques, including cloze tests @xcite and contrastive evaluation via minimal pairs @xcite . @xcite have created a rule-based translation system that converts english into various dialects. they use this system to conduct stress tests on multiple downstream models and reveal performance disparities between english dialects. methods to improve lms' robustness towards dialects include integrating morphological informa-tion into lms' tokenizers through inflection perturbations @xcite , manipulating the parse tree of the source sentence to align with the word order in the target dialect @xcite , and character noise injection @xcite . using perturbed data during lm pre-training or adapter training has shown significant benefits for dialectal variants of the glue tasks @xcite specifically designed to dialects @xcite . a related line of research concentrates on the processing of spoken dialects, with a specific emphasis on dialectal speech to standard language recognition @xcite and spoken dialect identification @xcite . ## perturbations based on dialect variations in this section, we introduce perturbations that are specifically motivated by dialectal variation. in english and german, these perturbations specifically focus on altering the morphosyntactic structure of the sentence to simulate dialects, while keeping the semantics unchanged. ## methodology we choose task-oriented dialogue systems as a task where we expect a high level of linguistic variation in real-life application settings. there is limited research on whether these systems commonly encounter inputs from dialect speakers in real-world applications @xcite . nevertheless, several works encourage the localization of dialogue systems to dialect varieties. one common motivational aspect shared by these works is the aim to encourage the use of dialects, with the expectation of positively impacting the prestige of the language @xcite . ## conclusion and future work this project tests the robustness of task-oriented dialogue systems (tod) towards english and german dialects. our methodology involves applying rule-based perturbations to translate tod datasets from standard american english to urban african american vernacular english, and from german to its non-standard variety. to the best of our knowledge, we are the first to design such perturbations for german. subsequently, we train multiple joined tod models, equipped with various transformerbased backbones, assessing their performance on intact and perturbed data. we conclude, that re rq 1: the impact of perturbed data on lm performance varies depending on the type of perturbation and the task. in general, we note a minor decrease in intent recognition but a notable drop in slot filling. issues in slot filling involve inaccuracies in boundary identification, mistakes in predicting slot types with altered word order, and frequent misalignments of slot labels with an extra verb. re rq 2: across languages, the performance drop varies by dataset and lm, indicating domain and language-specific patterns in response to phenomena-based perturbations. re rq 3: there is no clear winner, but mdeberta and rembert outperform other lms by gaining higher performance scores and being more robust to dialectal variations. future work includes (i) extension to other languages with distinct dialectal variation; (ii) development of fair evaluation approaches, that do not favor standard languages but account for dialects; (iii) incorporating phonological phenomena for a deeper understanding of dialectal variations in written and spoken forms. conducting similar experiments with other languages and dialects can help in understanding how these models generalize across diverse linguistic landscapes.