TK - KNN : A Balanced Distance-Based Pseudo Labeling Approach for Semi-Supervised Intent Classification (2023)

Abstract: The ability to detect intent in dialogue systems has become increasingly important in modern technology. These systems often generate a large amount of unlabeled data, and manually labeling this data requires substantial human effort. Semi-supervised methods attempt to remedy this cost by using a model trained on a few labeled examples and then by assigning pseudo-labels to further a subset of unlabeled examples that has a model prediction confidence higher than a certain threshold. However, one particularly perilous consequence of these methods is the risk of picking an imbalanced set of examples across classes, which could lead to poor labels. In the present work, we describe Top-K K-Nearest Neighbor (TK-KNN), which uses a more robust pseudo-labeling approach based on distance in the embedding space while maintaining a balanced set of pseudo-labeled examples across classes through a ranking-based approach. Experiments on several datasets show that TK-KNN outperforms existing models, particularly when labeled data is scarce on popular datasets such as CLINC150 and Banking77.

## introduction. large language models like bert @xcite have significantly pushed the boundaries of natural language understanding (nlu) and created interesting applications such as automatic ticket resolution @xcite . a key component of such systems is a virtual agent's ability to understand a user's intent to respond appropriately. successful implementation and deployment of models for these systems require a large amount of labeled data to be effective. although deployment of these systems often generate a large amount of data that could be used for fine-tuning, the cost of labeling this data is high. semi-supervised learning methodologies are an ob- unlabeled set of examples our tk-knn selection strategy existing threshold selection strategy imbalanced decision boundary balanced decision boundary @xcite . in this toy scenario, we chose @xmath0, where each class is represented by a unique shape. as the threshold selection strategy pseudo-labels data elements (shown as yellow) that exceed the confidence level, the model tends to become biased towards classes that are easier to predict. this bias causes a cascade of mislabels that leads to even more bias towards the majority class. indeed, research most closely realted to the present work is the slot-list model by @xcite , which focuses on the meta-learning aspect of semi-supervised learning rather than using unlabeled data. in a similar vein the gan-bert @xcite model shows that using an adversarial learning regime can be devised to ensure that the extracted bert features are similar amongst the unlabeled and the labeled data sets and substantially boost classification performance. other methods have investigated how data augmen-tation can be applied to the nlp domain to enforce consistency in the models @xcite , and several other methods have been proposed from the computer vision community. however, a recent empirical study found that many of these methods do not provide the same benefit to nlu tasks as they provide to computer vision tasks @xcite and can even hinder performance. intent classification remains a challenging problem for multiple reasons. generally, the number of intents a system must consider is relatively large, with sixty classes or more. on top of that, most queries consists of only a short sentence or two. this forces models to need many examples in order to learn nuance between different intents within the same domain. in the semi-supervised setting, many methods set a confidence threshold for the model and assign pseudo-labels to the unlabeled data if their confidence is above the threshold @xcite . this strategy permits high-confidence pseudo-labeled data elements to be included in the training set, which typically results in performance gains. unfortunately, this approach also causes the model to become overconfident for classes that are easier to predict. in the present work, we describe the top-k k-nearest neighbor (tk-knn) method for training semi-supervised models. the main idea of this method is illustrated in figure 1 . tk-knn makes two improvements over other pseudo-labeling approaches. first, to address the model overconfidence problem, we use a top-k sampling strategy when assigning pseudo-labels to enforce a balanced set of classes by taking the top-k predictions per class, not simply the predictions that exceed a confidence threshold overall predictions. furthermore, when selecting the top-k examples the sampling strategy does not simply rely on the model's predictions, which tend to be noisy. instead we leverage the embedding space of the labeled and unlabeled examples to find those with similar embeddings and combine them with the models' predictions. experiments using standard performance metrics of intent classification are performed on three datasets: clinc150 @xcite , banking77 @xcite , and hwu64 @xcite . we find that the tk-knn method outperforms existing methods in all scenarios and performs exceptionally well in the low-data scenarios. ## related work intent classification the task of intent classification has attracted much attention in recent years due to the increasing use of virtual customer service agents. recent research into intent classification systems has mainly focused on learning out of distribution data @xcite @xcite . these techniques configure their experiments to learn from a reduced number of the classes and treat the remaining classes as out-of-distribution during testing. although this research is indeed important in its own regard, it deviates from the present work's focus on semi-supervised learning. pseudo labeling pseudo labeling is a mainstay in semi-supervised learning @xcite @xcite . in simple terms, pseudo labelling uses the model itself to acquire hard labels for each of the unlabeled data elements. this is achieved by taking the argmax of the models' output and treating the resulting label as the example's label. in this learning regime, the hard labels are assigned to the unlabeled examples without considering the confidence of the model's predictions. these pseudo-labeled examples are then combined with the labeled data to train the model iteratively. the model is then expected to iteratively improve until convergence. the main drawback of this method is that mislabeled data elements early in training can severely degrade the performance of the system. a common practice to help alleviate mislabeled samples is to use a threshold τ to ensure that only high-quality (i.e., confident) labels are retained @xcite . the addition of confidence restrictions into the training process @xcite has shown improvements but also restricts the data used at inference time and introduces the confidence threshold value as yet another hyperparameter that needs to be tuned. another major drawback of this selection method is that the model can become very biased towards the easy classes in the early iterations of learning @xcite . recent methods, such as flexmatch @xcite , have discussed this problem at length and attempted to address this issue with a curriculum learning paradigm that allows each class to have its own threshold. these thresholds tend to be higher for majority classes lower for less-common classes. however, this only serves to exacerbate the problem because the lesscommon classes will have less-confident labels. a previous work by @xcite proposes a similar class balancing parameter to be learned per class, but is applied to the task of unsupervised domain adaptation. a close previous work to ours is co-training @xcite ] that iteratively adds a single example from each class throughout the self-training. another more recent work @xcite also proposes a balanced sampling mechanism for self-training, but starts from a zero-shot perspective and limits to two cycles of self-training. another pertinent work is by @xcite , who introduce contrastnet, a framework that leverages contrastive learning for few-shot text classification. this is particularly relevant to our study considering the challenges posed by datasets with a scarce number of labeled examples per class. a notable work by @xcite employs a contrastive learning-enhanced nearest neighbor mechanism for multi-label text classification, which bears some resemblance to the knn strategies discussed in our work. the tk-knn strategy described in the present work addresses these issues by learning the decision boundaries for all classes in a balanced way while still giving preference to accurate labels by considering the proximity between the labeled and the unlabeled examples in the embedding space. distance-based pseudo labeling another direction explored in recent work is to consider the smoothness and clustering assumptions found in semi-supervised learning @xcite for pseudo labeling. the smoothness assumption states that if two points lie in a high-density region their outputs should be the same. the clustering assumption similarly states that if points are in the same cluster, they are likely from the same class. recent work by @xcite propose a training-free approach to detect corrupted labels. they use a kstyle approach to detect corrupted labels that share similar features. the results of this work show that the smoothness and clustering assumptions are also applicable in a latent embedding space and therefore data elements that are close in the latent space are likely to share the same clean label. two other recent works have made use of these assumptions in semi-supervised learning to improve their pseudo-labeling process. first, @xcite use the wasserstein dis-tance to match clusters of unlabeled examples to labeled clusters for pseudo-labeling. second, the aptly-named feature affinity based pseudo-labeling @xcite ] method uses the cosine similarity between unlabeled examples and cluster centers that have been discovered for each class. the selected pseudo label is determined based on the highest similarity score calculated for the unlabeled example. results from both of these works demonstrate that distance-based pseudo-labeling strategies yield significant improvements over previous methods. however, both of these methods depend on clusters formed from the labeled data. in the intent classification task considered in the current study, the datasets sometimes have an extremely limited number of labeled examples per class, with instances where there is only one labeled example per class. this scarcity of labeled data makes forming reliable clusters quite challenging. therefore, the tk-knn model described in the present work adapted the k-nearest neighbors search strategy to help guide our pseudo-labeling process. ## results results from these experiments are shown in these results demonstrate the benefits of tk-knn's balanced sampling strategy and its use of the distances in the latent space. ## conclusions this paper introduces tk-knn, a balanced distance-based pseudo-labeling approach for semisupervised intent classification. tk-knn deviates from previous pseudo-labeling methods as it does not rely on a threshold to select the samples. instead, we show that a balanced approach that takes the model prediction and k-nearest neighbor similarity measure allows for more robust decision boundaries to be learned. experiments on three popular intent classification datasets, clinc150, banking77, and hwu64, demonstrate that our method improved performance in all scenarios.