Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Filling (2021)

Abstract: Predicting user intent and detecting the corresponding slots from text are two key problems in Natural Language Understanding (NLU). Since annotated datasets are only available for a handful of languages, our work focuses particularly on a zero-shot scenario where the target language is unseen during training. In the context of zero-shot learning, this task is typically approached using representations from pre-trained multilingual language models such as mBERT or by fine-tuning on data automatically translated into the target language. We propose a novel method which augments monolingual source data using multilingual code-switching via random translations, to enhance generalizability of large multilingual language models when fine-tuning them for downstream tasks. Experiments on the MultiATIS++ benchmark show that our method leads to an average improvement of +4.2% in accuracy for the intent task and +1.8% in F1 for the slot-filling task over the state-of-the-art across 8 typologically diverse languages. We also study the impact of code-switching into different families of languages on downstream performance. Furthermore, we present an application of our method for crisis informatics using a new human-annotated tweet dataset of slot filling in English and Haitian Creole, collected during the Haiti earthquake.

## introduction a cross-lingual setting is typically described as a scenario in which a model trained for a particular task in one source language (e.g. english) should be able to generalize well to a different target language (e.g. japanese). while semi-supervised solutions @xcite , inter alia) assume some target language data or translators are available, a zero-shot solution @xcite @xcite assumes none is available at training time. having models that generalize well even to unseen languages is crucial for tackling real world problems such as extracting relevant information during a new disaster @xcite or detecting hate speech @xcite , where the target language might be of low-resource or unknown. intent prediction and slot filling are two nlu tasks, usually solved jointly, which learn to model the intent (sentence-level) and slot (word-level) labels. such models are currently used extensively for goal-oriented dialogue systems, such as amazon's alexa, apple's siri, google assistant, and microsoft's cortana. finding the 'intent' behind the user's query and identifying relevant 'slots' in the sentence to engage in a dialogue are essential for effective conversational assistance. for example, users might want to 'play music' given the slot labels 'year' and 'artist' @xcite , or they may want to 'book a flight' given the 'airport' and 'locations' slot labels @xcite . a strong correlation between the two tasks has made jointly trained models successful @xcite @xcite . in a cross-lingual setting, the model should be able to learn this joint task in one language and transfer knowledge to another @xcite @xcite . this is the premise of our work. highly effective transformer-based multilingual models such as mbert @xcite and xlm-r @xcite have found success across several multilingual tasks in recent years. in the zero-shot cross-lingual transfer setting with an unknown target language, a typical solution is to use pre-trained transformer models and fine-tune to the downstream task using the monolingual source data @xcite . however, @xcite showed that existing transformer-based represenin the above code-switching example, the chunks are in chinese, punjabi, spanish, english, arabic, and russian. 'atis_airfare' represents an intent class where the user seeks price of a ticket. further, we show how code-switching with languages from different language families impacts the model's performance on individual target languages, even finding some counter-intuitive results. for instance, training on data code-switched between english and sino-tibetan languages is as helpful for hindi (an indo-aryan indo-european language) as code-switching with other indo-aryan languages, and turkic languages can be helpful for both chinese and japanese. ## methodology this section describes our problem definition, codeswitching algorithm, language families, and the training methodology. ## experimental setup we use the traditional cross-lingual task setting where each experiment consists of a source language and a target language. a model is trained on the source data (english) and evaluated on the target data (8 other languages). model on the validation set. consistent with the metrics reported for intent prediction and slot filling evaluation in the past, we also accuracy for intent and micro f1 foot_4 to measure slot performance. ## results & discussion effect of multilingual code-switching. the runtime of the models in in the translate-train (upper bound) scenario, it is not immediately clear if augmentation helps, since data in the same language as the target are always preferable to other language or code-switched data. at a minimum, augmentation does not hinder upper-bound performance ( for both intent and slot performance, the chunklevel model remained robust across the languages. for intent, the difference between word-level and sentence-level was insignificant. for slot, sentencelevel was in par with chunk-level on average. thus, evaluation on disaster dataset. we found that disaster data is more challenging than the atis dataset for transfer learning in nlu. the predictive performance is shown in ## conclusion & future work our study shows that augmenting the monolingual input data with multilingual code-switching via random translations at the chunk-level helps a zeroshot model to be language neutral when evaluated on unseen languages. this approach enhanced the generalizability of pre-trained language models such as mbert when fine-tuning for downstream tasks of intent detection and slot filling. additionally, we presented an application of this method using a new annotated dataset of disaster tweets. further, we studied code-switching with language families and their impact on specific target languages. addressing code-switching with language families during the pre-training phase and releasing a larger dataset of annotated disaster tweets in more languages are planned for future work.