Label efficient semi-supervised conversational intent classification (2023)

Abstract: To provide a convenient shopping experience and to answer user queries at scale, conversational platforms are essential for e-commerce. The user queries can be pre-purchase questions, such as product specifications and delivery time related, or post-purchase queries, such as exchange and return. A chatbot should be able to understand and answer a variety of such queries to help users with relevant information. One of the important modules in the chatbot is automated intent identification, i.e., understanding the user’s intention from the query text. Due to non-English speaking users interacting with the chatbot, we often get a significant percentage of code mix queries and queries with grammatical errors, which makes the problem more challenging. This paper proposes a simple yet competent Semi-Supervised Learning (SSL) approach for label-efficient intent classification. We use a small labeled corpus and relatively larger unlabeled query data to train a transformer model. For training the model with labeled data, we explore supervised MixUp data augmentation. To train with unlabeled data, we explore label consistency with dropout noise. We experiment with different pre-trained transformer architectures, such as BERT and sentence-BERT. Experimental results demonstrate that the proposed approach significantly improves over the supervised baseline, even with a limited labeled set. A variant of the model is currently deployed in production.

## introduction an automated conversational chatbot is essential to provide a seamless shopping experience and answer product-related questions at scale. an effective chatbot can assist and answer pre-purchase queries such as product specifications, offers, discounts, delivery time, and stock availability, as well as post-purchase queries such as exchange and return. due to users from diverse backgrounds interacting with the chatbot and minimizing a human agent transfer, a chatbot should be able to understand and handle a variety of user queries. one of the important ml components in the chatbot is automated intent identification, i.e., understanding the user's intention from the query text. post the correct intent identification, an appropriate dialog-flow can be initiated. an incorrect intent prediction negatively affects the dialog-flow and, hence the overall user experience. further, due to non-english speakers interacting with the chatbot, we observe a significant percentage of codemix hinglish queries ( 30%) and queries with grammatical errors, making intent detection even more challenging. training a supervised intent classification model under such a scenario would require a large amount of manually tagged data. however, due to internetscale operations, we have unlabeled query data available in a relatively large volume. this paper proposes a simple yet competent semi-supervised learning (ssl) approach for label-efficient intent classification. ssl has been proven effective in leveraging unlabeled data when only a small labeled set is available. specifically, we train a transformer bert model on a small labeled corpus along with a larger unlabeled query data. starting with limited labeled queries, we explore supervised as well as unsupervised data augmentation techniques. for the supervised data augmentation, we explore mixup @xcite and simple label preserving nlp augmentations @xcite . for training with unlabeled data, typically, ssl algorithms rely on an extra smoothness constraint which enforces the model to make consistent predictions on an unlabeled sample and its slightly perturbed version. moreover, it is observed that the type of noise/perturbation plays an important role and a trivial noise may not provide desired improvements @xcite . recently, a simple noise such as dropout has shown promising results for contrastive learning @xcite . we explore label consistency loss with dropout noise to train the bert model with unlabeled data. the model is trained with the linear combination of supervised and unsupervised loss components. one of the challenges with a limited labeled set is how to halt the training when the validation set is not available; otherwise, it may result in over-fitting. in our experiments, we perform the model updates till the training loss is converged. interestingly, training with dropout label consistency loss is less prone to over-fitting even with no validation set. we also noticed that the choice of label consistency loss has a prominent effect on the accuracy. for warm starting the training, we experiment with pre-trained bert and sentence-bert architectures. experimental results demonstrate that, over the supervised baseline, the intent classification accuracy can be boosted significantly with the proposed semi-supervised approach. unsupervised data augmentation (uda) @xcite has shown promising results for learning with unlabeled data along with a small labeled corpus. the idea is to enforce label consistency between two augmentations of the unlabeled sample. the authors also point out that the type of augmentation used significantly affects the accuracy of the model, and a trivial augmentation (such as adding gaussian noise) may not lead to desired improvements. recently, a contrastive learning approach that uses dropout noise has been shown to work well for self-supervised learning with textual data @xcite . since dropout is inher-ently present in pre-trained transformer models, this provides a simple yet efficient method for data augmentation. interpolation consistency training (ict) @xcite is a computationally efficient approach to train the model with ssl. ict encourages the prediction at an interpolation of unlabeled points to be consistent with the interpolation of the predictions at those points. for classification problems, ict moves the decision boundary to low density regions of the data distribution. for the supervised classification, mixup has been found to be an effective data augmentation technique @xcite . mixup is performed in the representation space for the text classification with transformers and is known to provide better regularization, and model calibration @xcite . ## proposed approach in this section, we describe details of the dataset, loss functions experimented with, and model training. ## ablation study in this section, we report ablation study results with different experimental settings. ## conclusion this paper proposes a simple yet competent semi-supervised learning approach for label-efficient conversational intent classification. we trained different transformer models with labeled as well as unlabeled data. we explored supervised mixup data augmentation for training with labeled samples, while for training with unlabeled samples, we experimented with label consistency loss with dropout. the results demonstrated that classification accuracy could be improved significantly over the supervised baseline with the proposed semi-supervised approach. specifically, sentence-bert was observed to perform better with a small number of labeled samples and even with code-mix hinglish queries. even without validation loss monitoring, it was noticed that training with dropout label consistency is less prone to over-fitting. through the ablation study, we studied the effect of the choice of label consistency loss and dropout probability on the accuracy. experimental results demonstrated the efficacy of the proposed approach. a variant of the model is currently deployed in production.