Industry Scale Semi-Supervised Learning for Natural Language Understanding (2021)

Abstract: This paper presents a production Semi-Supervised Learning (SSL) pipeline based on the student-teacher framework, which leverages millions of unlabeled examples to improve Natural Language Understanding (NLU) tasks. We investigate two questions related to the use of unlabeled data in production SSL context: 1) how to select samples from a huge unlabeled data pool that are beneficial for SSL training, and 2) how does the selected data affect the performance of different state-of-the-art SSL techniques. We compare four widely used SSL techniques, Pseudo-label (PL), Knowledge Distillation (KD), Virtual Adversarial Training (VAT) and Cross-View Training (CVT) in conjunction with two data selection methods including committee-based selection and submodular optimization based selection. We further examine the benefits and drawbacks of these techniques when applied to intent classification (IC) and named entity recognition (NER) tasks, and provide guidelines specifying when each of these methods might be beneficial to improve large scale NLU systems.

## introduction voice-assistants with speech and natural language understanding (nlu) are becoming increasingly prevalent in every day life. these systems, such as google now, alexa, or siri, are able to respond to queries pertaining multiple domains (e.g., music, weather). an nlu system commonly consists of an intent classifier (ic) and named entity recognizer (ner). it takes text input from an automatic speech recognizer and predicts intent and entities. for example, if a user asks "play lady gaga", the ic classifies the query to intent of playmusic, and the ner classifies "lady gaga" as artist. an important requirement for voice-assistants is the ability * equal contribution to continuously add support for new functionalities, i.e., new intents, or new entity types, while improving recognition accuracy for the existing ones. having high quality labeled data is the key to achieve this goal. however, obtaining human annotation is an expensive and time-consuming process. semi-supervised learning (ssl) provides a framework for utilizing large amount of unlabeled data when obtaining labels is expensive @xcite @xcite . ssl techniques have been shown to improve deep models performance across different machine learning tasks, including text classification, sequence labeling, machine translation and image classification @xcite @xcite @xcite . a common practice to evaluate ssl algorithms is to take an existing labeled dataset and only use a small fraction of training data as labeled data, while treating the rest of the data as unlabeled dataset. such evaluation, often constrained to the cases when labeled data is scarce, raises questions about the usefulness of different ssl algorithms in a real-world setting @xcite . in voice assistants, we face additional challenges while applying ssl techniques at scale including (1) how much unlabeled data should we use for ssl and how to select unlabeled data from a large pool of unlabeled data? (2) most ssl benchmarks make the assumption that unlabeled datasets come from the same distribution as the labeled datasets. this assumption is often violated as, by design, the labeled training datasets also contain synthetic data, crowd-sourced data to represent anticipated usages of a functionality, and unlabeled data often contain a lot of out of domain data. (3) unlike widely used nlu datasets such as snips @xcite , atis @xcite , real-world voice assistant datasets are much larger and have a lot of redundancy because some queries such as "turn on lights" might be much more frequent than others. due to such evaluation concerns, performance of different ssl techniques in "real-world" nlu applications is still in question. to address these issues, we study three data selection methods to select unlabeled data and evaluate how the selected data affect the performance of different ssl methods on a real-world nlu dataset in the english language. this paper provides three contributions: (1) design of a production ssl pipeline which can be used to intelligently select unlabeled data to train ssl models (2) experimental comparison of four ssl techniques including, pseudo-label, knowledge distillation, cross-view training, and virtual adversarial training in a real-world setting using data from amazon alexa (3) operational recommendations for nlp practitioners who would like to employ ssl in production setting. ## background semi-supervised learning techniques are capable of providing large improvements in model performance with little effort, which could play a crucial role in large scale systems in industry. in supervised learning, given a labeled dataset d l composed of input-label pairs (x, y), the goal is to learn a prediction model f θ (x), with parameters θ, that is able to predict the correct label y corresponding to a new unseen input instance x . ssl techniques aim to leverage an unlabeled dataset, d u , to create better performing models than those that could be obtained by only using d l . the two widely used ssl methods are: pseudo-label (pl), and knowledge distillation (kd). in pl, a teacher model trained on labeled data is used to produce pseudo-labels for the unlabeled data set. a student model trained on the union of the labeled and pseudo-labeled data sets, often outperforms the teacher model. @xcite . on the other hand, kd ssl methods do not assign a particular label to an unlabeled instance, but instead consider the whole distribution over the label space @xcite @xcite . in kd, it is hypothesized that leveraging the probability distribution over all labels provides more information than assuming a definitive label belonging to one particular class @xcite . in addition to pl and kd, virtual adversarial training (vat) and cross-view training (cvt) have achieved state-of-the-art ssl performance on various tasks including text classification, named entity recognition, and dependency parsing @xcite @xcite . in this paper, we conduct comprehensive experiments and analysis related to these commonly used ssl techniques, and discuss their pros and cons in the industry setting. data selection for ssl has been explored for different tasks including image classification @xcite , ner @xcite . model confidence based data selection is a widely used technique for ssl data selection where unlabeled data is selected on the basis of a classifier's confidence. due to the abundance of unlabeled data in production voice-assistants, model confidence based filtering leads to a very large data pool. to overcome this issue, we study different data selection algorithm which can further reduce the size of unlabeled data. ## methods we are interested in studying two different questions relevant to the use of unlabeled data in production environments: 1) how to effectively select ssl data from a large pool of unlabeled data, and 2) how do ssl techniques perform in realistic scenarios? to do so, we focus on the tasks of intent classification (ic) and named entity recognition (ner), two important components in nlu systems. the model architecture we study is an lstmbased multi-task model for ic and ner tasks, where we use 300-dimension fasttext word embeddings @xcite , trained on a large voice assistant corpus. foot_0 a shared 256-dimension bi-lstm encoder and two separate task-specific bi-lstm encoders (256-dimension) are applied to encode the sentences. a softmax layer and a conditional random field (crf) layer are used to produce predictions for ic and ner, respectively. below we describe our implementation of the ssl techniques and the data selection methods studied. ## data sets the main motivation of our study is to evaluate different data selection and ssl techniques in a production scale setting where we have a large amount of unlabeled data. to understand impact of data selection, we create two benchmark datasets for our experiments. in both experiments, using the pipeline shown in figure 1 , we first select m utterances from a very large pool of unlabeled data, and then apply intelligent data selection to further select n unlabeled utterances. commercial dataset: our commercial dataset provides an experimental setup to compare ssl techniques where labeled training data and unlabeled data come from a similar distribution. we choose four representative domains (i.e., categories for which the user can make requests) from a commercially available voice-assistant system for english language. the four selected categories are 1) communication: queries related to call, messages, 2) music: queries related to playing music, 3) notifications: queries related to alarms, timers, and 4) todos: queries related to task organization. for each domain, nlu task is to identify the intent (ic), and the entities (ner) in the utterance. for each domain, our dataset contains 50k unique training, 50k unique testing utterances, and hundreds of millions of utterances of unlabeled data. since, we do not know in advance to which domain each unlabeled utterance belongs, we first select 500k unlabeled utterance per domain to form their respective unlabeled data pool, using a domain classifier, as shown in figure 1 . the choice of 500k size is based on a series of kd based ssl experiments in music domain, with the ssl data pool size varying from 50k to 1m. it is observed that increasing ssl pool size beyond 500k starts to reduce the performance gain from ssl ( ## recommendations based on our empirical results, we make the following recommendations for industry scale nlu ssl systems. prefer vat and cvt ssl techniques over pl and kl: when selecting ssl techniques, cvt usually performs better for classification task while vat is preferable for ner task. in general, we would recommend vat since its performance in classification task is comparable to cvt and also because vat excels in ner task which is usually harder to achieve performance gain. use data selection to select a subset of unlabeled data: for industry setting where the volume of unlabeled data is impractically large, we introduce a data filtering pipeline to first reduce the size of unlabeled data pool to a manageable size. our experiments show that both submodular as well as committee based data selection could further improve ssl performance. we recommend submodular optimization based data selection in light of its lower cost and similar performance to committee based method. from experiments on snips data sets, we observe that further data selection does not bring extra improvement comparing to random selection. optimizing data selection, when unlabeled data pool is of a drastically different distribution from the labeled data, remains a challenge and could benefit from further research. ## conclusion in this paper, we conduct extensive experiments and in-depth analysis of different ssl techniques applied to industry scale nlu tasks. industrial settings come with some unique challenges such as massive unlabeled data with a mixture of in domain and out of domain data. in order to overcome these challenges, we also investigate different data selection approaches including submodular optimization and committee based filtering. our paper provides insights on how to build an efficient and accurate nlu system, utilizing ssl, from different perspectives (e.g. model accuracy, amount of data, training time and cost, etc). by sharing these insights with larger nlp community, we hope that these guideline will be useful for researchers and practitioner who aim to improve nlu systems while minimizing human annotation effort.