Federated Learning for Spoken Language Understanding (2020)

Abstract: Recently, spoken language understanding (SLU) has attracted extensive research interests, and various SLU datasets have been proposed to promote the development. However, most of the existing methods focus on a single individual dataset, the efforts to improve the robustness of models and obtain better performance by combining the merits of various datasets are not well studied. In this paper, we argue that if these SLU datasets are considered together, different knowledge from different datasets could be learned jointly, and there are high chances to promote the performance of each dataset. At the same time, we further attempt to prevent data leakage when unifying multiple datasets which, arguably, is more useful in an industry setting. To this end, we propose a federated learning framework, which could unify various types of datasets as well as tasks to learn and fuse various types of knowledge, i.e., text representations, from different datasets and tasks, without the sharing of downstream task data. The fused text representations merge useful features from different SLU datasets and tasks and are thus much more powerful than the original text representations alone in individual tasks. At last, in order to provide multi-granularity text representations for our framework, we propose a novel Multi-view Encoder (MV-Encoder) as the backbone of our federated learning framework. Experiments on two SLU benchmark datasets, including two tasks (intention detection and slot filling) and federated learning settings (horizontal federated learning, vertical federated learning and federated transfer learning), demonstrate the effectiveness and universality of our approach. Specifically, we are able to get 1.53% improvement on the intent detection metric accuracy. And we could also boost the performance of a strong baseline by up to 5.29% on the slot filling metric F1. Furthermore, by leveraging BERT as an additional encoder, we establish new state-of-the-art results on SNIPS and ATIS datasets, where we get 99.33% and 98.28% in terms of accuracy on intent detection task as well as 97.20% and 96.41% in terms of F1 score on slot filling task, respectively.

## introduction in recent years, spoken language understanding (slu) technology, which typically involves intent detection and slot filling, plays a crucial part in goal-oriented dialogue systems. the intent detection is treated as an utterance classification problem, which can be modeled using various classifiers based on deep neural networks @xcite @xcite . the slot filling task can be formulated as a sequence labeling problem, and one of the popular approaches with good performance is using crf and cnn @xcite . the slots represent the word-level information while the intent stands for the sentence-level information @xcite . these two tasks are normally considered as parallel tasks but may have cross-impact on each other @xcite . despite the impressive results, most of the existing frameworks focus on a single individual dataset and are directly trained on the specific individual target dataset. the data-specific model may well suffer from overfitting when the data in the downstream dataset is scarce. in addition, because the design and training of the model are data-specific, it is difficult for the model to transfer the knowledge learned from the target dataset to the other dataset directly, where the two datasets may be completely different. it means that a well-trained model on the target dataset may have poor performance on the other dataset, which indicates the current models suffer from poor robustness. in short, we argue that for the slu task, studies used to model on a single individual dataset are not friendly to the model robustness. intuitively, learning from multiple datasets at the same time will improve the performance and robustness of the model, compared to the single dataset training. thus, it is reasonable to combine multiple datasets for modeling in slu. a multi-task learning framework, which is used for combining multiple tasks in the previous works @xcite , may have the potential to unify multiple datasets, though, it may have data leakage problems, which is unacceptable for collaborative task training between multiple institutions. what's more, the multi-task framework is a special case of one of our federated learning framework, i.e., vertical federated learning, which will be discussed in the following paragraph. recently, @xcite , @xcite and @xcite propose an emerging artificial intelligence technology, i.e., federated learning, whose goal is to train a high quality centralized model based on datasets that are distributed across multiple clients without sharing the clients' data. inspired by the success of federated learning, we introduce a federated learning framework to perform the slu task. in implementations, we treat each sub-task as a client and propose an mv-encoder as the centralized model. the motivation for proposing the mv-encoder into our federated learning framework stems from the fact that a key goal in the slu task is to effectively utilize the potential linguistic knowledge of the sentence and capture multi-granularity text representations for the input text. thus, inspired by @xcite , we attempt to obtain multi-granularity text representations for our framework. to this end, our mv-encoder consists of four sub-encoders, i.e., position-wise encoder, local encoder, global encoder and time series encoder. finally, we conduct extensive experiments on three federated learning settings, i.e., horizontal federated learning, vertical federated learning and federated transfer learning, and two benchmark datasets, i.e., snips and atis, to validate the effectiveness and the universality of our approach. moreover, bert @xcite , a powerful pre-trained language model, is employed to further boost the performance of our framework. overall, the main contributions of our work are as follows: ## approach in this section, we focus on the basic module of our architecture. firstly, we introduce an mv-encoder module to extract the multi-granularity sequence information from several views, which is shown in figure 1 . then, we describe the intent decoder and slot decoder in the decoding stage. finally, we introduce the implementation of three federated learning settings and the corresponding datasets on our experiments. the architecture of our federated learning framework is shown in figure 2 . ## experiments in this section, we first describe two benchmark datasets and experimental settings for intent detection and slot filling tasks, and their evaluation metrics. next, we describe some previous methods and list their performance. finally, we present our evaluations of the proposed framework on three federated learning ## analysis significant improvements among two metrics have been witnessed on both two publicly datasets in section 4. however, we would like to know the reason for the improvement. in this section, we first study the effect of our proposed mv-encoder. next, we explore the effect of the federated learning framework. finally, we study the effect of bert representation in our framework. ## conclusion in this paper, we propose a federated learning framework to perform slu, where the centralized model is realized by the proposed mv-encoder to capture the multi-granularity information of the input sentences. our framework can unify various types of knowledge, i.e., text representations, from different datasets and tasks, without the sharing of downstream task data. furthermore, we implement our framework on three federated learning settings and conduct extensive experiments on two benchmarks datasets, i.e., snips and atis. the experimental results and analysis demonstrate the effectiveness of our framework and the mv-encoder. specifically, our approach brings noticeable improvements compared to the baseline. in particular, by leveraging bert as an additional encoder, we achieve the best performances on snips and atis datasets, where we get 99.33% and 98.28% in terms of accuracy on the intent detection task as well as 97.20% and 96.41% in terms of f1 score on the slot filling task, respectively.