Exploring Description-Augmented Dataless Intent Classification (2024)

Abstract: In this work, we introduce several schemes to leverage description-augmented embedding similarity for dataless intent classification using current state-of-the-art (SOTA) text embedding models. We report results of our methods on four commonly used intent classification datasets and compare against previous works of a similar nature. Our work shows promising results for dataless classification scaling to a large number of unseen intents. We show competitive results and significant improvements (+6.12% Avg.) over strong zero-shot baselines, all without training on labelled or task-specific data. Furthermore, we provide qualitative error analysis of the shortfalls of this methodology to help guide future research in this area.

## introduction task-oriented dialogue systems (tods) by design, aid the user in accomplishing tasks within specific domains, and can have a wide range of applications from shopping @xcite to healthcare @xcite . modular tods @xcite will typically contain an intent classification component @xcite @xcite used by a dialogue manager to determine the appropriate task the user intends to complete. in recent years, neural-based models using supervised training have reached state-of-the-art on many natural language processing tasks, including intent classification. however, supervised learning methods require human-labelled data for a predefined set of intents, which may be time-consuming and labour-intensive to acquire @xcite , and may have poor scalability if new intents are added, or task definition changed. an early approach to tackle this problem is dataless intent classification @xcite which aimed to leverage the pairwise similarities between semantic representations of utterances and intent classes to perform classification without reliance on human-labelled data. however, this approach relies heavily on the quality of semantic representations @xcite . in recent years, successful zero-shot intent classification approaches @xcite @xcite have received greater attention, whereby learning conducted using labelled examples of a subset of seen intent labels is transferred to unseen intents. however, these methods still require human-labelled data, and tend to bias towards seen intents, with the number of unseen intents also generally much lower than seen intents @xcite . in this work, with the significant recent advancements in the quality of text embedding models @xcite , we explore the potential for dataless intent classification methods using a number of recent state-of-the-art text embedding models. we introduce several approaches for generating intermediate textual representations for intents, most notably using intent label descriptions, and formalise our methodology. we perform extensive evaluation of our methods, including scenarios with large numbers of intents from different domains, using three commonly used intent classification datasets. we summarise our contributions as follows: ## analysis and future work in-domain saturation we visualise the embeddings generated by our best-performing model (bge large ) on the 4 evaluation datasets using t-sne (van der @xcite , along with the embedding for the intent label description to gain insight into the source of errors in our approach. figure 1 shows the distribution of embeddings on the atis and snips datasets. in the interest of space, visualisations of clinc and massive are shown in appendix g . we observe a poor alignment on the atis dataset between the intent label descriptions (figure 1a ) and utterance embeddings corresponding to each class, possibly explaining the poor performance in general on this dataset across models. we note the single-domain nature of the atis dataset, with all utterances relating to air-travel/flight; additionally, we note the significantly imbalanced nature of the atis dataset @xcite , with ∼ 74% of utterances belonging to the flight class, which is a label that overlaps the domain of the dataset. we hypothesise this may lead to the intent label descriptions being much worse at capturing semantic information distinct to each class. this is supported by analysis of the pairwise embedding similarities of utterances ## conclusion dataless classification allows for scaling to a large number of unseen classes without requiring training on labelled, task-specific data. the benefits of such an approach can enhance development of task-oriented dialogue systems in application to data-poor or compute-limited scenarios where supported intents may also change as the system is developed. in this paper, we have explored the potential of current sota text embedding models in dataless intent classification settings using three different approaches for representing intent classes and compared our results against strong zero-shot learning baselines. we proposed a method for standardising the generation of intent label descriptions with an aim to minimise the amount of human annotation required to further support scaling to high numbers of intent classes. our results have shown that description-augmented dataless classification methods can achieve comparable, and sometimes superior performance to zero-shot methods on the task of intent classification.