On the cross-lingual transferability of multilingual prototypical models across NLU tasks (2021)

Abstract: Supervised deep learning-based approaches have been applied to task-oriented dialog and have proven to be effective for limited domain and language applications when a sufficient number of training examples are available. In practice, these approaches suffer from the drawbacks of domain-driven design and under-resourced languages. Domain and language models are supposed to grow and change as the problem space evolves. On one hand, research on transfer learning has demonstrated the cross-lingual ability of multilingual Transformers-based models to learn semantically rich representations. On the other, in addition to the above approaches, meta-learning have enabled the development of task and language learning algorithms capable of far generalization. Through this context, this article proposes to investigate the cross-lingual transferability of using synergistically few-shot learning with prototypical neural networks and multilingual Transformers-based models. Experiments in natural language understanding tasks on MultiATIS++ corpus shows that our approach substantially improves the observed transfer learning performances between the low and the high resource languages. More generally our approach confirms that the meaningful latent space learned in a given language can be can be generalized to unseen and under-resourced ones using meta-learning.

## introduction traditionally, natural language understanding (nlu) is an intermediate module between the user interface and the dialogue management module in a dialogue system. it aims to extract semantic information from a user's query or utterance to fill slots in a domain specific semantic frame. domain classification, intent detection and slot filling are three core components belonging to the nlu. they are in charge of determining the domain or service of a users query, its underlying goal or intent and asso-ciating utterance segments with conceptual labels, called slots, similar to named entity recognition. nlu is usually defined as a supervised learning problem, involving conventional machine learning models on massive amount of annotated training data, which are language dependent. this prerequisite has prevented its widespread adoption for poorly endowed languages and for small technology companies that do not benefit from millions of users to gather data. besides the requirement of a large amount of annotated data being available, domains, intents and slots are language dependent. consequently, in practice, the resulting systems are hardly adaptable to expand to new languages. as a solution to this problem, cross-lingual transfer approaches were developed to leverage the knowledge from well-resourced languages, with task specific data available to under-resourced languages with little or no data. recent efforts focused on training transformer models multilingually such as the multilingual version of bert @xcite . while earlier work demonstrated the effectiveness of multilingual models to learn representations which are transferable across languages, they show limitations when applied to low-resource languages @xcite . from another perspective lowshot learning such as few-shot and zero-shot, aims to transfer knowledge learned from one language to another when the training data is limited or is missing some task labels. as a core contribution, we explore the potential for cross-lingual transferability of multilingual transformer-based model @xcite (mbert) combined with a few-shot learning algorithm based on prototypical representations. we also introduce a zero-shot scenario, where models are trained on multiple languages and evaluated on another. our proposed approach relies on appending a mbert encoder module to the prototypical neural network, which is a proven few-shot model, originally designed for image classification. our experimental results show that the generated model trained with a limited number of annotated training examples outperforms the transfer learning based approach on multiatis++ dataset @xcite and can be applied to unseen languages directly with decent performance. ## related work the availability of large datasets has enabled deep learning methods to achieve great success in a variety of fields. however, most of these successes are based on supervised learning approaches, which require lots of labeled data to train. most datasets are only available in english. only a few other languages are supported, and most of them are considered as under-resourced languages. recently, meta-learning approaches have enabled the development of task-agnostic learning algorithms capable of far generalizations (crossdomain or cross-lingual) in the context of having a low-data regime. because literature on low-shot learning is vast and diverse, only the most relevant approaches to this work are presented and we refer the reader to @xcite and @xcite for a surveys of earlier work. ## approach in this section, we present the design of a prototypical neural network and its episodic training procedure before introducing our approach. ## conclusions in this paper, we demonstrate the opportunities in leveraging mbert-based modeling using few-shot learning for both intent classification and slot filling tasks on under-resource languages. we found that our approach model is a highly effective technique for training models for low-resource languages. this illustrates the performance gains that can be achieved by exploiting language interrelationships learnt with transfer learning, a conclusion further emphasised by the fact that multilingual results outperformed other configuration models (target only and specifically multilingual zero-shot) regardless of the approach. overall, pnns models outperform mbert-based transfer learning approach, enabling us to train competitive nlu systems for underresources languages with only a fraction of training examples. from this work a new challenge naturally comes up and a possible direction is to adapt a few-shot setting to a joint approach of intent detection and slot filling, like in zhang and wang (2016), liu and lane (2016) and zhang et al. (2019) , which demonstrates that performing these two tasks jointly improves the performance of both of them.