Few-shot Pseudo-Labeling for Intent Detection (2020)

Abstract: In this paper, we introduce a state-of-the-art pseudo-labeling technique for few-shot intent detection. We devise a folding/unfolding hierarchical clustering algorithm which assigns weighted pseudo-labels to unlabeled user utterances. We show that our two-step method yields significant improvement over existing solutions. This performance is achieved on multiple intent detection datasets, even in more challenging situations where the number of classes is large or when the dataset is highly imbalanced. Moreover, we confirm this results on the more general text classification task. We also demonstrate that our approach nicely complements existing solutions, thereby providing an even stronger state-of-the-art ensemble method.

## introduction labeling user utterances as intents is of paramount importance for chatbots. intent classes are usually engineered by a linguistic team, and these classes are likely to evolve with time. the most frequent situations include new unanticipated intents found by dialogue traces analysis, the need to introduce new intents that were previously handled using web forms, or the urge to split existing intents into several new ones for fine-grained user interactions. regardless the rationale, each time the intent referential evolves, a significant amount of new labeled data is required to train a decent intent detection model. this calls for a time-consuming and error-prone labeling process -it is therefore expensive. consequently, intent detection systems need robust models in few-shot settings in order to avoid to repeatedly suffer from labeling user utterances into intents. a solution to this problem is to generate pseudo-labels for unlabeled utterances. a pseudo-label is a label (intent class) automatically assigned to an unlabeled utterance using the knowledge found in the set of labeled data. the resulting dataset composed of both labeled and pseudo-labeled utterances is then used to train a supervised intent detection model. most pseudo-labeling methods combine advances in word representations @xcite @xcite and then consider unsupervised clustering algorithms @xcite in order to propagate known labels to unlabeled data (see section 2). nonetheless, there are major drawbacks to existing pseudo-labeling algorithms. first, since clustering algorithms are used to partition user utterances representations, it leads to provide a pseudo-label to all unlabeled data. this is error-prone since cluster shapes for the entire dataset are unknown and hardly predictable in a few shot setting. consequently, it is much harder to learn a discriminate hyperplane when training the downstream intent detection models since some data are incorrectly labeled. worse still, this grows exponentially as the number of intents (classes) increases in practice since it is harder to predict correct pseudo-labels, and the hyperplane is more difficult to learn since it requires more discriminative power. a corollary is also that confidence values in pseudo-labels is hard to produce: since cluster shapes are unknown in advance, distances to centroids cannot reliably be used as a factor a confidence in a pseudo-label. in computer vision, the same issues exist for the image classification task and therefore end-to-end systems have been proposed -prototypical networks @xcite . applied to intent detection, there are also part of related work and baselines in this work. as contribution, we first present a new folding/unfolding hierarchical clustering algorithm for pseudolabeling, which can be considered as a dynamic hierarchical clustering algorithm (section 3). this original algorithm, by nature, is able to predict a pseudo-label only when it has sufficient clues (thereby reducing labeling noise) and to provide a pseudo-label normalized confidence score based on a temperature weighting mechanism -when training the downstream intent detection model using both labeled and pseudo-labeled utterances, those weights are used by penalizing the loss (section 3.3). in order to validate our pseudo-label approach, we evaluate its impact on the downstream task of intent detection using four datasets and fine-tuned bert-based language models (section 4). overall, in few-shot configurations and even when the number of classes is large, our proposed method is able to positively impact the intent or text classification tasks accuracies (section 5), beyond the state-of-the-art. the experiments also exhibits a robustness of our methods to imbalance bias, which is common in practice. finally, section 6 concludes. ## related works a growing amount of work has been done on the conversational ai domain @xcite over the recent years. there are different types of conversational agents: question-answering systems @xcite , end-to-end conversational models @xcite , or task-oriented bots @xcite . in the latter case, a central part of the process is the understanding of user utterances, a special case of text classification. convincing approaches to this problem rely on neural networks (lee and dernoncourt, 2016), thus requiring large labeled datasets, which are costly to obtain. in practice, as more users interact with the conversational agent, it is easy to collect a large amount of raw data. having access to a small labeled dataset along with a large unlabeled one calls for semi-supervised learning solutions @xcite for which label propagation is of paramount importance. one approach to this problem is self-training @xcite , where the model trains itself for multiple iterations, each time adding its most confident prediction to the training data. this approach has multiple weaknesses, the main one being the fact that the model cannot correct its own mistakes. one way of dealing with such a problem is to use tri-training (zhi-hua @xcite , a framework where three models are trained iteratively. at each step, common predictions of two models are added to the training set of a third. this continues until all models reach a stationary state. a substantial body of works focus on the few-shot classification problem, that is handling labeled data scarcity in the downstream training phase exclusively @xcite . in our work, we tackle the problem differently: we propagate the knowledge we have on this small amount of data to unlabeled data points, before feeding both truly labeled and pseudo-labeled data to the intent detection system. this intermediate step is of the utmost practical interest: firstly, propagating knowledge on unlabeled points helps us understand how the data is structured. secondly, in a labeling framework, it allows us to select interesting samples for our in-house linguists to label or correct given low pseudo-label confidence foot_1 as a human-in-the-loop approach, instead of selecting samples randomly. the more accurate the pseudo-labels, the more efficient it is for linguists. from a broader perspective, it is worth noticing that semi-supervised learning (learning using both labeled and unlabeled data) has also received a lot of attention in computer vision @xcite , where annotations are time-consuming and error-prone as well. to overcome this problem, imagespecific features -like coloration, pixel-level values -are used @xcite . given a distance matrix between data points, labels from known samples are propagated to previously unlabeled data. unfortunately this is hardly applicable in nlp as complex features are hardly intrinsic to words but rather based on distributional semantics. indeed, in nlp we usually rely on pre-trained word embeddings @xcite and build sentence representations by averaging word vectors @xcite or contextualized word embeddings @xcite . from these sentence representations, labels from known utterances can be propagated mainly using clustering strategies. the first and most simple available method is the naive nearest neighbor algorithm -henceforth nknn -that is for each unlabeled data point and each class, we compute its average similarity to the labeled samples of this class. then, we assign to this point the label for which this average similarity is maximal, mapping all unlabeled data points to a pseudo-label. the methods described so far are two-fold: unsupervised pseudo-labeling is followed by classification semi-supervised learning. conversely, it is possible to learn end-to-end few shot classifiers, that are classifiers learning to classify using only few shots, without pseudo-labels. the state-of-the-art for endto-end few-shot classification @xcite is an extension of prototypical networks @xcite . while originally designed for computer vision @xcite , such networks have recently gained traction in the field of few-shot setups in natural language processing @xcite . even though these methods are not meant to produce pseudo-labels directly, they excel at classifying utterances using only a few labeled samples. in this method, for each label, a prototype is built as the average embedding of support samples. then, during training, query samples are classified using their distance to prototypes of each label. the optimizer's goal is to make this distance the lowest possible. it is therefore interesting to consider as related and as a later baseline, a system that is based on prototypical networks @xcite where the input is the bert @xcite sentence encoder. this approach will hereafter be referred to as proto. in our experiments (section 5), we also include the state-of-the-art result for end-to-end few shot classification introduced by @xcite . in this variant, unlabeled samples are used during an additional step, where the prototypes are refined using a soft k-means technique. in our experiments, this baseline will be denoted as proto++. ## two-fold pseudo-labeling in this section, we will describe our folding/unfolding pseudo-labeling method (section 3.1), as well as the aggregated approach (section 3.2). we also introduce a loss weighting scheme (section 3.3), which will be able to mitigate the impact of incorrect pseudo-labels on the intent detection downstream task. we advocate that this solution is state-of-the-art (section 5) yet simple, therefore very valuable. ## experimental setup alongside with our contribution, the systems under evaluation are nknn, proto/proto++, as introduced in section 2. we implemented the nknn baseline, and for the prototypical-based approaches, we use the code associated to the seminal prototypical network paper foot_3 , using bert @xcite as our sentence encoder. for the proto++ baseline, to refine prototypes, we use 20 unlabeled samples per class both at training and testing time, as @xcite showed that it yielded better performances. as suggested by @xcite , we do not fine-tune prototypical networks at test time, as it is reported to decrease performances. moreover, since prototypical networks are an end-to-end approach, it is first impossible to fine-tune such networks using a single shot for each class, and second, the embeddings of utterances for all labels are fine-tuned in the networks themselves. the evaluation of the different systems is two-fold: we first assess the relevance of pseudo-labels retrieved from these methods in different few-shot situations, and then we evaluate the performances of the intent detection model (see section 4.4) trained using those pseudo-labels alongside with the few labeled data. in order to measure the various performances, we use the weighted f1-score in all our experiments. ## results and analysis in snips on the snips dataset, even though our folding/unfolding approach outperforms all competitors, those results must be taken lightly. as we can see, with only 5 shots, we are able to recover pseudolabels and build an intent detection model with a 0.988 f1-score. those huge performances clearly hint that this dataset is not challenging at all: the t-sne projection for this dataset in figure 2 supports this claim. this means that we can hardly make concluding remarks on how the systems relatively perform on more realistic datasets. overall, the massive usage of snips to evaluate intent detection techniques in the literature @xcite becomes open to question. oos on the oos dataset, our method largely comes out on top, both on pseudo-label and intent detection results. on the intent detection part, our method is better than the aggregated ones, showing that other methods penalize the aggregation. this finding is very important, as it proves that using an ensemble of various methods do not always lead to better performances. concerning the prototypical baselines, this is the dataset where the proto++ is the further away from its former version, proto. this comes from the fact that this dataset has the highest number of classes: when doing a soft-kmeans step, if the number of classes is high, then it is as much noise for the prototype refinement. results on this dataset prove that our method is not handicapped when using a large number of classes, making it more robust to real-life scenarios. additionally, we see that removing the bert-fit+nknn from the aggregation yields better results. this poor performance was not to be seen in the pseudo-label results, as this baseline was performing second behind our approach. this raises the importance of using this double experiment, and confirms that better pseudo-labels do not directly imply a better model in the downstream task of intent detection. liu as explained in section 4.2, this dataset is very imbalanced. still, we get the same results as the two previous datasets, which were more balanced: our method is able to come out on top on both pseudolabeling as well as intent detection results. additionally, the ablation study where we withdraw our method from the aggregation yields the worst results in almost every situation, hinting that we contribute the most to this aggregation. this shows that our method is robust to various situations, including the case of imbalanced datasets. r8 on the r8 dataset, even though we now deal with text classification and much longer sentences, we have similar results as on others intent detection datasets. our method comes out on top, even achieving better performances than aggregated methods during the classification step. this shows that our folding/unfolding method is robust, and not only suited for intent detection tasks, but also for the more general subject of text classification. ## conclusion in this paper we introduced a new state-of-the-art, hierarchical clustering inspired method for pseudolabeling intent detection datasets. this is of the utmost practical interest in order to help linguists to faster provide new intent detection model as the intent classes repository evolve with time. our method, being hyper-parameter-free, is robust to various situations, even when the number of classes is high. overall, it is straightforward to implement and still the best option available for pseudo-labeling user utterances in few-shot classification. through an ablation study, we also show that our method is complementary to other baselines so that it contributes to aggregated approaches significantly. moreover, we demonstrate that while proto++ approach is known to out-perform its former version proto on image classification, this does not apply for learning few-shot intent detection models. finally, we showed that if we continue to evaluate intent detection using very easy -yet popular -datasets like snips, we are ultimately not solving the actual downstream task in practice, so we encourage any intent detection researchers and practitioners to use it with caution.