  Building the Natural Language Understanding (NLU) modules of task-oriented
Spoken Dialogue Systems (SDS) involves a definition of intents and entities,
collection of task-relevant data, annotating the data with intents and
entities, and then repeating the same process over and over again for adding
any functionality/enhancement to the SDS. In this work, we have developed an
Intent Bulk Labeling system for SDS developers. The users can interactively
label and augment training data from unlabeled utterance corpora using advanced
clustering and visual labeling methods. We extend the Deep Aligned Clustering
work with a better backbone BERT model, explore techniques to select the seed
data for labeling, and develop a data balancing method using an oversampling
technique that utilizes paraphrasing models. We also look at the effect of data
augmentation on the clustering process. Our results show that we can achieve
over 10% gain in clustering accuracy on some datasets using the combination of
the above techniques. Finally, we extract utterance embeddings from the
clustering model and plot the data to interactively bulk label the data,
reducing the time and effort for data labeling of the whole dataset
significantly.
