Semi-supervised Interactive Intent Labeling (2021)

Abstract: Building the Natural Language Understanding (NLU) modules of task-oriented Spoken Dialogue Systems (SDS) involves a definition of intents and entities, collection of task-relevant data, annotating the data with intents and entities, and then repeating the same process over and over again for adding any functionality/enhancement to the SDS. In this work, we showcase an Intent Bulk Labeling system where SDS developers can interactively label and augment training data from unlabeled utterance corpora using advanced clustering and visual labeling methods. We extend the Deep Aligned Clustering work with a better backbone BERT model, explore techniques to select the seed data for labeling, and develop a data balancing method using an oversampling technique that utilizes paraphrasing models. We also look at the effect of data augmentation on the clustering process. Our results show that we can achieve over 10% gain in clustering accuracy on some datasets using the combination of the above techniques. Finally, we extract utterance embeddings from the clustering model and plot the data to interactively bulk label the samples, reducing the time and effort for data labeling of the whole dataset significantly.

## introduction acquiring an accurately labeled corpus is necessary for training machine learning (ml) models in various classification applications. labeling is an expensive and labor-intensive activity requiring annotators to understand the domain well and to label the instances one at a time. in this work, we explore the task of labeling multiple intents visually with the help of a semi-supervised clustering algorithm. the clustering algorithm helps learn an embedding representation of the training data that is well-suited for downstream labeling. in order to label, we further reduce the high dimensional representation using the umap @xcite . since utterances are short, uncovering their semantic meaning to group them together is very challenging. sbert @xcite showed that out-of-the-box bert @xcite ) maps sentences to a vector space that is not very suitable to be used with common measures like cosine-similarity and euclidean distances. this happens because in the bert network, there is no independent sentence embedding computation, which makes it difficult to derive sentence embeddings. researchers utilize the mean pooling of word embeddings as an approximate measure of the sentence embedding. however, results show that this practice yields inappropriate sentence embeddings that are often worse than averaging glove embeddings @xcite . many researchers have developed sentence embedding methods: skip-thought @xcite , in-fersent @xcite , use @xcite , sbert @xcite . state-of-the-art sbert adds a pooling operation to the output of bert to derive a fixed-sized sentence embedding and fine-tunes a siamese network on the sentence-pairs from the nli @xcite and stsb @xcite datasets. the deep aligned clustering (dac) @xcite introduced an effective method for clustering and discovering new intents. dac transfers the prior knowledge of a limited number of known intents and incorporates a technique to align cluster centroids in successive training epochs. the limited known intents are used to pre-train the model. the authors use the pre-trained bert model @xcite to extract deep intent features, then pre-train the model with a randomly selected subset of labeled data. the pre-trained parameters are used to obtain well-initialized intent representations. k-means clustering is performed on the extracted intent features along with a method to estimate the number of clusters and the alignment strategy to obtain the final cluster assignments. the k-means algorithm selects cluster centroids that minimize the euclidean distance within the cluster. due to this euclidean distance optimization, clustering using the sbert model to extract feature embeddings naturally outperforms other embedding methods. in our work, we have extended the dac algorithm with the sbert as an embedding backbone for clustering of utterances. in semi-supervised learning, the seed set is selected using a sampling strategy: "a simple random sample of size n consists of n individuals from the population chosen such that every set of n individuals has an equal chance to be the sample actually selected." @xcite . however, these sample subsets may not represent the original data adequately because randomization methods do not exploit the correlations in the original population. in a stratified random sample, the population is classified first into groups (called strata) with similar characteristics. then a simple random sample is chosen from each strata separately. these simple random samples are combined to form the overall sample. stratified sampling can help ensure that there are enough observations within each strata to make meaningful inferences. dac uses the random sampling method for seed selection. in this work, we have explored a couple of stratified sampling approaches for seed selection in hope to mitigate the limitations of random sampling and improve the clustering outcome. another issue we address in this work is class sample imbalance. seed selection generally yields an imbalanced dataset, which in turn impairs the predictive capability of the classification algorithms @xcite . some methods manipulate the training data, aiming to change the class distribution towards a more balanced one by undersampling or oversampling @xcite . @xcite is a popular oversampling technique proposed to improve random oversampling. in one variant of smote, borderline minority instances are heuristically selected and linearly interpolated to create synthetic samples. in this work, we take inspiration from the smote method and choose borderline minority instances and paraphrase them using a sequence to sequence paraphrasing model. the paraphrases provide natural and meaningful augmentations of the dataset that are not synthetic. previous work has shown that data augmentation can boost performance on text classification tasks @xcite @xcite . @xcite used neural machine translation (nmt) @xcite to translate the non-english side of the parallel text to get english-english paraphrase pairs. this method has been scaled to generate large paraphrase corpora @xcite . prior work in learning paraphrases has used autoencoders @xcite , encoder-decoder architectures as in bart @xcite , and other learning frameworks such as nmt @xcite . data augmentation using paraphrasing is a simple yet effective strategy that we explored in this work to improve the clustering. for interactive visual labeling of utterances, we build up from the learnt embedding representation of the data and fine-tune it using the clustering. dac learns to cluster with a weak self-supervised signal to update its representation and to optimize both local (via k-means) and global information (via cluster alignment). this results in an optimized intent-level feature representation. this high dimensional latent representation can be reduced to 2-3 dimensions using the uniform manifold approximation and projection (umap) @xcite . we use rasa whatlies foot_0 library @xcite to extract the umap embeddings. for interactive labeling, we utilize an interactive visualization library called human learn foot_1 @xcite ) that allows us to draw decision boundaries on a plot. by building on top of the work of rasa bulk labelling foot_2 ui @xcite , we augment the interface with our learnt representation for interactive labeling. although we focus on nlu, other studies like 'conversation learner' @xcite focus on interactive dialogue managers (dm) with human-in-the-loop annotations of dialogue data via machine teaching. note also that although the majority of task-oriented sds still involves defining intents/entities, there are recent examples that argue for a richer target representation than the classical intent/entity model, such as smcalflow @xcite . ## methodology figure 1 describes the semi-supervised labeling process. we start with the unlabeled utterance corpus and apply seed sampling methods to select a small subset of the corpus. once the selected subset is manually labeled, we address the data imbalance with our paraphrase-based minority oversampling method. we can also augment the labeled corpus with paraphrasing to provide more data for the clustering process. the dac algorithm is applied with improved embeddings to extract the utterance representation for interactive labeling. ## experimental results to conduct our experiments, we use the bank-ing @xcite and clinc @xcite datasets similar to the dac work @xcite . we also use another dataset called kidspace that includes utterances from a multimodal learning application for 5-to-8 years-old children @xcite . we hope to utilize this system to label future utterances into relevant intents.