SIDLR : Slot and Intent Detection Models for Low-Resource Language Varieties (2023)

Abstract: Intent detection and slot filling are two critical tasks in spoken and natural language understandingfor task-oriented dialog systems. In this work, we describe our participation in slot and intent detection for low-resource language varieties (SID4LR) (Aepli et al., 2023). We investigate the slot and intent detection (SID) tasks using a wide range of models and settings. Given the recent success of multitask promptedfinetuning of the large language models, we also test the generalization capability of the recent encoder-decoder model mT0 (Muennighoff et al., 2022) on new tasks (i.e., SID) in languages they have never intentionally seen. We show that our best model outperforms the baseline by a large margin (up to +30 F1 points) in both SID tasks.

## introduction digital conversational assistants have become increasingly pervasive. examples of popular virtual assistants include siri, alexa, and google. a crucial factor in the effectiveness of these systems is their capacity to understand user input and respond or act accordingly to fulfill particular requirements. most of these applications are voice-based and hence need spoken language understanding (slu). slu typically starts with automatic speech recognition (asr), taking the sound of spoken language and transcribing it into text. then, it handles natural language understanding (nlu) tasks to extract semantic features from the text including question answering, dialogue management, intent detection, and slot filling. the intent detection task aims to recognize the speaker's desired outcome from a given utterance. and slot filling focuses on identifying the main arguments or the spans of words in the utterance that contain semantic information relevant to the intent. in this work, we present our participation in the slot and intent detection for low-resource language varieties (sid4lr; @xcite ) shared task. the shared task takes as its target three low resources languages-swiss german (gsw), south tyrolean (st), and neapolitan (nap). the main objective of the sid4lr shared task is to find the most effective approach for transferring knowledge to less commonly spoken languages that have limited resources and lack a standard writing system, in the zero-shot setting (i.e., without use of any training data). in the context of the shared task, we target the following four main research questions: q1: can successful models on english sid tasks be generalizable to new unseen languages (i.e., the zero-shot setting)? q2: how do models trained on a language from the given language family fare on a low-resource variety from the same family under the zeroshot setting (i.e., with no access to training data from these low-resource varieties). for example, in our case, we ask how do models trained on german perform on swiss german or south tyrolean, and how do models trained on italian perform on neapolitan. q3: what impact does exploiting data augmentation techniques such as paraphrasing and machine translation have on the sid tasks in the zero-shot context? q4: are the existing large multilingual models, trained using multitask-prompted fine-tuning, able to achieve zero-shot generalization to sid tasks in languages that they have never intentionally seen? the rest of this paper is organized as follows: section 2 is a literature review on intent and slot detection tasks. the shared task, the source data provided in sid4lr, and the external parallel data we exploit to build our models are described in section 3. in section 4, we provide information about datasets, baselines, and data preprocessing. the baseline, and multilingual pre-trained language models we used are described in section 5. we present our experimental settings and our training procedures in section 6. section 7 is an analysis and discussion of our results. and we conclude in section 8. ## related work the problem of low-resource slot and intent detection for languages with limited training data has been the focus of several recent research works. in this section, we discuss some of the most relevant and recent works, including datasets, benchmarks, and models that aim to address this challenge. ## sid4lr shared task task formulation. intent detection and slot-filling are critical nlp tasks where, given an utterance, a system is responsible for parsing the user's intent and extracting relevant information to act or reply appropriately. while many neural-based models have achieved sota performance for these tasks, their success often depends on large amounts of labeled data. however, many real-world datasets are limited to specific domains and are only available in english or a few other languages. as a result, it is important to reuse existing data in high-resource languages to develop models for low-resource lan-guages, especially since tasks like intent classification and slot-filling require abundant labeled data. shared task problem statement. this shared task of sid aims to address the challenges of performing sid for low-resource language varieties for the following languages: swiss german, south tyrolean, and neapolitan. the training data provided consists of the cross-lingual slot and intent detection (xsid 0.4 ) corpus (van der goot et al., 2021), a cross-lingual spoken language understanding dataset, covering 12 languages (arabic, chinese, dutch, danish, english, german, indonesian, italian, japanese, kazakh, serbian, turkish) from six language families with english training. the task allowed the use of pre-trained models and external data including data from the target language. evaluation metric. the primary evaluation metric for slot filling is the span f 1 score, where both span and label must match exactly, and accuracy is used to evaluate intent detection where it is calculated through the ratio of the number of correct predictions of intent to the total number of sentences. more details regarding the shared task can be found in @xcite . we converted the dataset into a json format that includes intents and slots. this json file was then converted to huggingface dataset format for easy use with our transformer models. a sample of the resulting json format is shown in listing 2. ## pre-trained language models in this study, we evaluate several popular multilingual transformer-based language models, including mbert, xlm-r, sbert, labse, laser, and mt0. these models are capable of effectively capturing cross-lingual embeddings, enabling transfer learning across multiple languages. below we provide a description of each model used in our experiments on the training dataset. mbert. is the multilingual version of bert @xcite , which is an encoder model with bidirectional representations from transformers trained with a denoising objective. mbert is trained on wikipedia for 104 languages including german and italian. xlm-r. @xcite ) is a transformerbased multilingual masked language model pretrained on more than 2tb of filtered common-crawl data in 100 languages, including languages including german and italian. xlm-r uses a transformer model @xcite trained with a multilingual masked language model xlm @xcite . sbert. sentence-bert (sbert) @xcite , is a modification of the pretrained bert @xcite model that uses siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. as we work under a multilingual context, we use the multilingual versions from previously monolingual sbert models @xcite which is trained for sentence embedding in 50+ languages from various language families. labse. language-agnostic bert sentence encoder (labse) @xcite ) is a bertbased model trained to generate sentence embeddings in 109 different languages. the model's pre-training approach involves a combination of masked language modeling and translation language modeling. the pre-training process combines masked language modeling with translation language modeling. labse is useful for producing sentence embeddings in multiple languages and performing bi-text retrieval. laser. language-agnostic sentence representations (laser) @xcite ) is a contextualized language model based on a bilstm encoder trained on parallel data from opus website (tiedemann, 2012) using a translation objective. the laser model can handle 200 different languages. mt0. @xcite is a group of sequence-to-sequence models that come with different sizes from 300m to 13b parameters trained to investigate the cross-lingual generalization through multitask finetuning. mt0 can execute human instructions in many languages without any prior training. the models are fine-tuned from preexisting mt5 @xcite multilingual language models using a cross-lingual task mixture called xp3. these refined models are capable of cross-lingual generalization to unseen languages. ## experiments and settings training on english data. as a baseline setting, we train all the pre-trained models described in section 5 on the english part of the multilingual dataset xsid 0.4 @xcite and evaluate them on swiss german, south tyrolean, and neapolitan under a zero-shot setting. training on german/italian data. our second approach aims to train all the pre-trained models on the language family of low-resource languages (i.e., german for swiss german and south tyrolean, and italian for neapolitan, respectively) under the zeroshot setting. so, we extract the german and italian sid data from xsid 0.4 , and then fine-tune all our models on both datasets. then, we evaluate the german models on gsw and st tasks and the italian models on the nap task. training on multilingual dataset. next, we explore a third training approach that involves the full multilingual xsid 0.4 dataset. to do so, we combine all the 12 available languages in the xsid 0.4 dataset and fine-tune our pre-trained models on this combined dataset. we then evaluate each target using a zero-shot setting. this approach allows us to train on larger and more diverse datasets. in total, we generate 502, 936 training sentences across all languages in the dataset. paraphrase and machine translation. to improve the performance of our pretrained models, we also explore the impact of data augmentation techniques such as paraphrasing and machine translation. specifically, we aime to examine how these techniques can enhance the performance of our models on cross-lingual slu tasks. to this end, we experiment with different data augmentation strategies, including paraphrasing and machine translation. paraphrasing is performed using the qualityguided controlled paraphrase generation (qcpg) model @xcite , resulting in a total of 130, 815 sentences in english. these sentences are then translated into german and italian using the opus-mt model @xcite , creating cross-lingual datasets for our experiment. to further augment our training data for lowresource languages, we leverage meta ai's no language left behind (nllb), which provides opensource models capable of high-quality translations between 200 languages (including low-resource languages @xcite ). to create our new training data using the nllb model, we first use fasttext to detect the language codes of our target languages. next, we utilize nllb models to translate the english training data into the predicted language codes. the language codes identified for our target languages are deu_latn for swiss german, est_latn for south tyrolean, and ita_latn for neapolitan. we generate 43, 605 sentences for each of the three languages. it is worth noting that we ensure that the labels for each sentence remain the same throughout the paraphrasing and machine translation process to maintain the integrity of the data. training on external data. since the language models we employ do not have a strong representation of the low-resource languages used on the task, we leverage large corpora of each of the lowresource languages into the training process. by incorporating external datasets, the models are exposed to more comprehensive information about the semantics of each low-resource language, enabling them to better capture the nuances and complexities of the target languages. training mt0 as discussed in section 5, the mt0 models share the same architecture as mt5/t5 models, i.e., they are encoder-decoder models. therefore, we train them for intent classification and slot detection using the data preprocessing approach described in section 4. we utilize the peft library provided by huggingface (sourab mangrulkar, 2022) to train the mt0-small, mt0-base, and mt0-large models. our approach involves using lora @xcite , which allows us to achieve sota performance while consuming significantly less memory. for the mt0-xxl models, we utilize deepspeed @xcite with cpu offloading to train a model with 13b parameters on a 40gb a100 gpu. combining models. in recent studies, joint learning techniques that combine multiple classification approaches have produced promising results @xcite . these approaches involve concatenating the outputs of individual models and passing the resulting output through multiple neural network layers, allowing the resulting network to be trained jointly. in this part of our experiments, we investigate the effectiveness of this approach in zero-shot settings by combining multilingual models. specifically, we combine laser embeddings, from the laser model, with other multilingual models including mbert, sbert, labse and xlmr. ## results and discussions our results on the validation data suggest that larger models generally achieve better performance, implying that higher parameter counts result in better cross-lingual and zero-shot setting performance. moreover, as the mt0 models are fine-tuned from pre-existing mt5 multilingual language models, they are capable of performing cross-lingual generalization on unseen languages. this capability may be a possible reason for the mt0 models outperforming other models in zero-shot settings. official shared task (test) results. our findings regarding the performance of larger models are also observed in the test set. it is worth noting that we use the validation set for model selection, which resulted in higher scores than those achieved on the test set. this is because @xcite . ls: laser @xcite +sbert @xcite . ll: laser+labse @xcite . lx: laser+xlm-r @xcite . underline: best-performing models for each setting. bold: best f 1 score across all the experiments and settings. the validation data is similar to the data used during training, while the test data is entirely new and unseen. as a result, the test scores may be lower due to differences in the distribution of data between the training and test sets. nevertheless, our mt0 models consistently outperform the baseline models on the test set, providing further evidence for the effectiveness of larger models in sid tasks. ## conclusion we described our contribution to the sid4lr @xcite shared tasks. our models target both the slot and intent sub-task in three proposed low-resource languages, namely, swiss german, south tyrolean, and neapolitan. we test the utility of existing pretrained language models such as mt0 @xcite on the intent detection and slot filling tasks. we show that such models can lead to improving the results of the baseline with an average of +27 f 1 points. in the future, we intend to use mt0 to jointly model the intent detection and slot filling tasks for improving overall performance.