To What Degree Can Language Borders Be Blurred In BERT -based Multilingual Spoken Language Understanding? (2020)

Abstract: This paper addresses the question as to what degree a BERT-based multilingual Spoken Language Understanding (SLU) model can transfer knowledge across languages. Through experiments we will show that, although it works substantially well even on distant language groups, there is still a gap to the ideal multilingual performance. In addition, we propose a novel BERT-based adversarial model architecture to learn language-shared and language-specific representations for multilingual SLU. Our experimental results prove that the proposed model is capable of narrowing the gap to the ideal multilingual performance.

## introduction recently, modern voice-controlled devices with virtual assistants like alexa, siri, and google assistant, have been working their way into every activity of our daily life from playing a song to driving a car. spoken language understanding (slu) models interpreting the semantic meaning conveyed by an user's spoken utterance, are the ai brains of these devices. the task of slu is usually divided into two sub-tasks, namely intent classification (ic), which identifies the intent of an user's utterance; and slot filling (sf), which extracts semantic constituents from the utterance. consider an annotated utterance from the atis dataset @xcite : the sf sub-task should classify "where" and "is" as o (i.e. other); and "mco" as bairport code, while the ic sub-task should identify city as the intent. current state-of-the-art slu models are mostly dnn-based joint models of the two sub-tasks @xcite @xcite . these models usually contain two individual decoders to detect slot and intent labels on top of a shared encoder. as in many other nlp fields, the bert encoder-decoder architecture has shown its strong ability in capturing contextual information from data sources to improve slu performance @xcite . for common slu use cases, the bert encoder is pre-trained on a huge amount of unlabeled texts and then fine-tuned on slu training data. since the voice-controlled device market has been expanding at an incredible rate all over the world, there is a rising need of fast language expansion for slu models. multilingual technology, which allows the development of a single model for multiple languages, and transferring knowledge from a data resource to languages other than its own, is currently one of the best solutions to meet the increased need. traditionally, an slu model is trained on supervised monolingual data leading to the fact that a virtual assistant often can use only one language within a working session. in contrast, a multilingual slu model trained on supervised multilingual data, can provide the virtual assistant the ability to talk in multiple languages within a working session. moreover, from a development perspective, multilingual modeling helps not only to reduce the number of models to build, but also to reduce supervised data needs by transferring knowledge across languages. however, as languages differ from each other in various aspects from lexicon to syntax, the cross-language knowledge transfer of the current multilingual techniques may still be limited. this raises the questions of: i) what an ideal cross-language knowledge transfer would be; ii) what a naive cross-language knowledge transfer would be; and iii) to what degree a multilingual model can transfer knowledge across languages. as one of the most successful multilingual techniques recently, multilingual bert (mbert) @xcite is being used more and more commonly in natural language processing models in both zero-shot foot_0 and multilingual foot_1 scenarios @xcite . since mbert is trained without any cross-lingual objectives and does not leverage aligned data, its strong cross-lingual abilities are surprising and have, in turn, spurred research aiming to understand why it is able to achieve them @xcite . in this paper, we, however, focus on addressing the question of to what degree it can blur the language borders in the multilingual slu setting by comparing its ability to transfer knowledge across languages with the ideal and naive cases. in addition, as mbert is pre-trained for general purpose, we are also interested in how to improve its multilinguality for the particular use case of multilingual slu. our contribution in this paper is two-fold: first, by performing a wide range of experiments with bilingual and trilingual slu models, we show that although mbert is substantially good in blurring language borders even on distant language groups, there is still a gap to the ideal multilingual performance. second, we propose a novel adversarial model architecture to learn language-shared and language-specific representations on top of mbert representations when fine-tuning on slu data. the experimental results prove that our proposed approach can narrow the gap to the ideal multilingual performance. ## related work as one of the most successful model architectures in natural language understanding recently, bert models have been explored in various researches for slu @xcite @xcite . in these works, slu is considered as a downstream task of a trained bert model. in particular, after being pre-trained on a large amount of unlabelled data, the bert embeddings and encoders are fine-tuned on supervised slu data together with two slu-adapted decoders for the ic and sf sub-tasks. due to the fast expansion of voice-controlled device market, there has been a rising interest in crosslingual and multilingual slu modeling. in the former direction, supervised data from one or multiple source languages is leveraged to improve the slu performance on a target language @xcite @xcite . in the later direction, slu models are trained on supervised multilingual data for multiple target languages @xcite . to obtain a multilingual model, @xcite simply initialized the embeddings and encoder of a bert-based slu model by mbert's parameters, and then trained the full model on a mix of supervised datasets from multiple languages. noticeably, the slu model used in @xcite was designed towards cross-lingual scenarios using a soft-alignment method to improve the slot projection between source and target languages. in this paper, we focus on only multilingual slu modeling. unlike @xcite , our adversarial model is designed specially for multilingual model building. the impressive abilities of mbert in cross-lingual and multilingual natural language understanding applications, have recently led to an increasing body of research analyzing how it is able to achieve those. for instance, @xcite probed the cross-linguality of mbert using zero-shot transfer learning on morphological and syntactic tasks and found that mbert is able to create multilingual representations. however, to the best of our knowledge, probing multilingual representation learning for downstream tasks, like slu, and addressing the question of how far it is from the ideal expectation, have not yet been explored in literature. adversarial approaches aiming to account for linguistic differences across languages by dividing the model into language-shared and language-specific representations, have been explored for the slu sub-tasks. recently, @xcite investigated the sub-tasks in isolation using bilstms and focused on improving slu for low-resource languages. meanwhile, @xcite explored bilstms to improve named entity recognition which is close to the sf sub-task of slu. in this work, instead of isolating the two sub-tasks, we propose a joint model based on mbert for multilingual use cases. ## bert-based multilingual slu models in this section, we describe the two multilingual slu models evaluated in this paper. chen et al. (2019a) and gaspers et al. (2020) , our model consists of a bert encoder receiving sub-word and position embedding layers as inputs, an ic decoder to identify intent labels, and an sf decoder to predict slot labels. in particular, given an utterance u, the final hidden states of the tokens h 1 . . . h t produced by the encoder, are used as the token representations. the sentence presentation, denoted by s u , is computed from h 1 . . . h t by using max-pooling. ## an ideal multilingual slu performance vs. a naive multilingual slu performance to address the question of to what degree knowledge can be transferred across languages in multilingual slu, in this section, we define an ideal and a naive multilingual performance. let us consider a multilingual slu model m for n languages l 1 , l 2 , . . . l n . we use the following notations: given d, the average performance of m on the target languages is computed in eq. 8. ## experiments in this section, we first describe the multilingual performance evaluation and experimental settings. afterwards, we present experiments on the resampled multiatis++ dataset and subsequently experiments with real-world-data. adversarial multilingual slu models as multi-lang. and lang.-adv., respectively. the ideal and naive baselines on each of the target languages are computed as in sec. 4 by using the bert-based model described in sec. 3.1 as the monolingual model m . ## conclusion we have addressed the question of to what degree language borders can be blurred in bert-based multilingual slu. our experimental results on a wide range of multilingual slu datasets showed that although mbert is substantially good in blurring language borders even on distant language groups, there is still a gap to the ideal multilingual performance. to narrow this gap, we proposed an adversarial model architecture which uses bert and cnn encoders to learn language-shared and language-specific representations for slu.