Exploiting In-Domain Bilingual Corpora for Zero-Shot Transfer Learning in NLU of Intra-Sentential Code-Switching Chatbot Interactions (2022)

Abstract: Code-switching (CS) is a very common phenomenon in regions with various co-existing languages. Since CS is such a frequent habit in informal communications, both spoken and written, it also arises naturally in Human-Machine Interactions. Therefore, in order for natural language understanding (NLU) not to be degraded, CS must be taken into account when developing chatbots. The co-existence of multiple languages in a single NLU model has become feasible with multilingual language representation models such as mBERT. In this paper, the efficacy of zero-shot cross-lingual transfer learning with mBERT for NLU is evaluated on a Basque-Spanish CS chatbot corpus, comparing the performance of NLU models trained using in-domain chatbot utterances in Basque and/or Spanish without CS. The results obtained indicate that training joint multi-intent classification and entity recognition models on both languages simultaneously achieves best performance, better capturing the CS patterns.

## introduction multilingual speakers outnumber monolingual speakers in the world @xcite . in regions with various coexisting languages, a common feature of natural interactions amongst speakers is the continuous casual switching between the concerned languages or "codes". this phenomenon, known as code-switching (cs), is very frequent in both spoken and written informal interactions @xcite . in fact, the percentage of cs in social networks ranges from 14.5% to 49.06% for the corpora explored in @xcite . moreover, in the study al-qaysi and al-emran that explores the educators and learners' attitudes towards using cs online, 86.40% of the students and 81% of the teachers claim to actively code-switch while chatting on social networks. as is to be expected, cs also arises spontaneously during human-machine interactions such as conversing with chatbots (i.e., conversational agents). this being the case, @xcite reveal that these interlocutors strongly prefer chatbots that do understand cs. the most common strategy employed by chatbots to understand the interlocutor is to employ intent and entity-based annotation schemata. this involves intent -communicative purpose-detection and entity -key words-classification processes @xcite . for example, in the utterance "i want to have an italian meal that does not exceed 15 euros", the intent behind the user's query is to order food given the entity labels cheap and italian. allowing multilingual interactions with chatbots involves running a language identifier prior to each speaker turn and executing language-specific natural language understanding (nlu) models. however, this approach is only effective for intersentential cs, where the code alternation happens at utterance boundaries. in the case of intrasentential cs, where the same utterance contains words or phrases belonging to two or more languages @xcite , detecting every intent and entity of the utterance poses a major challenge for existing algorithms @xcite . the previous sentence with intra-sentential cs would be "quiero an italian meal que no supere los 15 euros" (spanish in italics). in this case, neither an nlu model in english nor an nlu model in spanish would detect that the intent is order food. one strategy to solve this problem would be treating cs as a language itself and training an nlu model with examples containing cs. however, collecting data with cs is quite challenging, as it hardly exists in written form and requires bilingual annotators. instead, obtaining labeled monolingual data in multiple languages is easier and zero-shot cross-lingual transfer learning (tl) has been proved to perform well across different natural language processing (nlp) tasks, including nlu @xcite . thus, this tl approach would be more practical to exploit in real-world industrial chatbot applications to be deployed in cs regions. unfortunately, such methodology has not yet been explored in the literature, mainly due to the scarcity of real intra-sentential cs datasets. in this paper, a basque-spanish cs corpus is exploited to evaluate different zero-shot cross-lingual tl experiments aiming to examine whether such multilingual training methodologies are capable of addressing the nlu problem of intra-sentential cs chatbot interactions. for this purpose, the effectiveness of three multilingual models is analysed in their ability to understand cs: one fine-tuned using a chatbot corpus in basque, another one fine-tuned on a corpus of spanish chatbot utterances and a third one fine-tuned using both corpora simultaneously. it is important to underline that none of the models was exposed to cs during training, and that the monolingual basque and spanish training corpora belong to the same domain of the cs corpus used for testing purposes. through this comparison it has been determined that models fine-tuned on indomain bilingual corpora simultaneously are able to generate cross-lingual bonds and perform better against cs. the fact that zero-shot fine-tuning of multilingual bert models on both monolingual languages enhances their effectiveness in understanding cs stands as one of the main contributions of this work. the remaining of the paper is structured as follows: section 2 reviews recent work in the area of joint intent and entity detection and multilingual models. section 3 analyzes the main characteristics of the corpora used both for training and evaluation purposes. section 4 presents the architecture and specifications of the joint intent and entity detection implementation employed. section 5 shows the results obtained in the different experiments carried out and, finally, section 6 highlights the main conclusions and proposes tentative lines for future work. ## implementation strategies the joint intent detection and entity classification nlu model developed in this work takes the one presented by @xcite and its corresponding implementation foot_0 as a baseline. for our experiments, the baseline model architecture has been adapted as shown in figure 1 to support the detection of multiple intents in the same utterance. this adaptation has involved two major changes. on the one hand, intent label representation has been adapted to allow assigning more than one intent to each utterance. for this purpose, one-hot foot_1 encoding has been adopted and consequently the activation function of the final layer has been changed from softmax to sigmoid. on the other hand, the loss function selected for multi-intent classification optimisation has been changed to binary crossentropy, as it allows each utterance to have more than one associated intent @xcite . this loss function predicts whether each possible intent appears in the utterance regardless of the rest of the intents. the loss function used to classify entities remains sparse categorical cross-entropy, since it meets the assumption that each token belongs to a single entity category. the goal of the implemented joint multi-intent and entity classifier is to minimise the sum of the two individual loss functions. the bert model employed is bert-base-multilingual-cased or mbert, which has 12 layers, 768 hidden states and 12 heads. and the best performing hyperparameters are a maximum length and batch size of 128; the adam optimiser (kingma and ba, 2015); a learning-rate of 9e-5; a dropout probability of 0.1; and 50 epochs.