Evaluating Cross-Lingual Transfer Learning Approaches in Multilingual Conversational Agent Models (2020)

Abstract: With the recent explosion in popularity of voice assistant devices, there is a growing interest in making them available to user populations in additional countries and languages. However, to provide the highest accuracy and best performance for specific user populations, most existing voice assistant models are developed individually for each region or language, which requires linear investment of effort. In this paper, we propose a general multilingual model framework for Natural Language Understanding (NLU) models, which can help bootstrap new language models faster and reduce the amount of effort required to develop each language separately. We explore how different deep learning architectures affect multilingual NLU model performance. Our experimental results show that these multilingual models can reach same or better performance compared to monolingual models across language-specific test data while require less effort in creating features and model maintenance.

## introduction the recent surge in popularity of voice assistants, such as google home, apple's siri, or amazon's alexa resulted in interest in scaling these products to more regions and languages. this means that all the components supporting spoken language understanding (slu) in these devices, such as automatic speech recognition (asr), natural language understanding (nlu), and entity resolution (er) are facing the challenges of scaling the development and maintenance processes for multiple languages and dialects. when a voice assistant is launched in a new locale, its underlying speech processing components are often developed specifically for the targeted country, marketplace, and the main language variant of that country. many people assume that if a device "understands" and "speaks" in a specific language, for example english, it should be able to work equally well for any english-speaking country, but this is a misunderstanding. for instance, if a speaker of uk english asks a device trained on data collected in the united states "tell me a famous football player", it is highly unlikely that this device will provide the user's desired answer, since football means different things in the us and uk cultures. as a result, developers need to take into account not only the language or dialectal differences, but also local culture, to provide the right information in the right language setup. an increase in the number of target marketplaces often means a linear increase in effort needed to develop and maintain such locale-specific models. nlu models, which classify the user's intent and extract any significant entities from the user's utterance, face the same challenge of maintaining high accuracy while being able to accommodate multiple dialects or language content. the major tasks in nlu are intent classification and slot filling. intent classification is a task to predict what action the user intends the voice assistant to take. slot filling is a task to identify the specific semantic arguments for the intention. for example, if the user's request is to "play poker face by lady gaga", the user's intention will be "play music", while in order to fulfill this command with specified details, the system needs to capture the slots for {song @xmath0, and {artist @xmath1. these tasks are called intent classification (ic) and named entity recognition (ner), respectively. one common approach is to use a max-entropy (maxent) classification model for the ic task and a conditional random fields (crf) model for the ner task. following the advent of deep learning techniques in related fields, such as computer vision and natural language processing, deep learning is becoming more popular in nlu as well. some of the recent multilingual approaches to nlu include, for example, the convolutional neural network (cnn) model for sentence classification @xcite , or the long short-term memory (lstm) model for ner prediction @xcite . in the deep neural network architecture, the aforementioned nlu tasks can be combined into a single multi-task classification model. an increasing number of experiments also focus on multilingual setups, especially in the field of machine translation, where the task is to translate input from one language to another @xcite . one recent thread of multilingual research centers around learning multilingual word representation. multilingual word embeddings in the shared cross-lingual vector space have one main property: words from different languages but with similar meaning must be geometrically close. this property allows for transfer learning from one language to another in various multilingual tasks, such as dependency parsing @xcite or classification and ner @xcite . a number of model architectures have been proposed to pre-train multilingual word representations, such as leveraging large-scaled lstm networks trained on monolingual corpora and adversarial setup for space alignment @xcite , or transformers trained on multilingual corpora as a single language model @xcite . although some of these models can be used to solve ic and ner tasks by appending corresponding decoders to generate final predictions, it is not straightforward to use them in production environments due to latency and memory constrains. a different way of benefitting from larger models could be to use them for transfer learning to smaller-size models to improve their performance by initializing some parts of the model with close-to-optimal rather than random weights. in this paper, we extend the multi-task approach studied in (do and gaspers, 2019) to a general multilingual model for ic and ner tasks, based on deep learning techniques, such as a bidirectional long short-term memory (bilstm) crf sequence labeling model for ner along with a multilayer perceptron (mlp) for ic. we also explore multilingual transfer learning and its benefits to our setup. transfer learning is widely adapted for zero-shot or few-shot setups, and was explored in some multilingual nlp studies @xcite @xcite , and also has been used in multi-task ic-ner models (do and gaspers, 2019), yet to the best of our knowledge, there is no study applying transfer learning for data-rich target languages in a multilingual setup. in our experiment, we apply few-shot transfer learning from data-rich languages to a language with a smaller amout of training data. in additon, we also apply transfer learning to mimic the situation of expanding the model ability to same-level-resource language with known context from another high-resource language(s), such that the new multilingual model will "inherit" context information from its ancestors. we investigate these approaches to transfer learning and their effects on model performance. we show that transfer learning can improve nlu model performance even in data-rich conditions. nlu model is first trained to recognize the utterance domain, such as music, weather, notifications, and then it is trained to perform domain-specific ic and ner tasks. in our experiments, the domain classification (dc) model is a maxent logistic regression model. despite the relatively simple architecture, this model shows good performance on dc task in data-rich conditions. for ic and ner tasks, we build a multi-task deep neural network (dnn) model. we first map input tokens into a share-space word embedding, and then feed them into a bilstm encoder to obtain context information; this content then propagates to the downstream tasks, with crf used for ner prediction, and an mlp classifier used for ic prediction. we call this a self-trained multilingual (stm) model. figure 1 shows the model architecture. figure 1 : multi-task model for ic and ner prediction. tokens are first embedded into word vectors, then fed in to a bilstm encoder, whose output is used in both crf and mlp for ner and ic, respectively. in the transfer learning experiments, we examined the following approaches: we limit our architecture to a small-sized dnn model to avoid memory and latency issues in production environment: ## conclusions in this paper, we propose a framework for building general multilingual nlu models, which can be used across different marketplaces and languages. to choose the model with the best performance, we use language-specific test sets to evaluate the candidate models and their corresponding baseline models (e.g. english bilstm-crf model vs. monolingual english maxent-crf model) along four metrics, domain accuracy, intent accuracy, slot f1, and frame accuracy. the models which win in most of the evaluation metrics are the final picks. we find that models built from a simple multi-task bilstm-crf model setup are comparable to standard production models in terms of latency constraints required for on-the-fly voice assistant conversational models. we observe performance improvements in all models with the introduction of transfer learning. encoder transfer produced the greatest improvements whereas the transfer of the decoder did not bring much change when compared to the baseline model performance, except when tested on an english test set, when the transfer learning is performed from the model trained on english data. this is due to the fact that the target non-english language contains slots or intents which are not included in the pre-trained model, thus the decoder fails to predict correct classes simply because they are missing in the vocabulary. to mitigate this effect, a decoder with default initialization gives better performance because it now can embrace all available slots and intents in the target language realm. furthermore, we find that a model pre-trained in a multilingual setup performs better than the one trained on a monolingual data set. this confirms that a multilingual model built based on lexically and orthographically similar languages may provide more beneficial context information to any similar target language. experimental result on hindi show that such a multilingual model can work even for non-alike languages with the same or better performance improvement. this confirms that a common multilingual model can be used to support multiple language with better results than a set of monolingual models. with a single general multilingual nlu model, bootstrapping new languages can be faster as we can use cross-lingual contextual information from all existing high-resource languages. at the same time, maintaining only one model requires much less effort in terms of regular model updates.