Exploring the Use of Natural Language Descriptions of Intents for Large Language Models in Zero-shot Intent Classification (2024)

Abstract: In task-oriented dialogue systems, intent classification is crucial for accurately understanding user queries and providing appropriate services. This study explores the use of intent descriptions with large language models for unseen domain intent classification. By examining the effects of description quality, quantity, and input length management, we identify practical guidelines for optimizing performance. Our experiments using FLAN-T5 3B demonstrate that 1) high-quality descriptions for both training and testing significantly improve accuracy, 2) diversity in training descriptions doesn’t greatly affect performance, and 3) off-the-shelf rankers selecting around ten intent options reduce input length without compromising performance. We emphasize that high-quality testing descriptions have a greater impact on accuracy than training descriptions. These findings provide practical guidelines for using intent descriptions with large language models to achieve effective and efficient intent classification in low-resource settings.

## introduction in task-oriented dialogue systems, mapping user utterances to a predefined set of intents is crucial and is known as 'intent classification.' this process is essential because it helps determine the service that the user requires, making it the foundational step in fulfilling the user's goal via a chatbot @xcite @xcite . due to the vast range of domains where chatbots can be utilized and the limited availability of intent classification data, research on transferring intent classifiers to unseen domains under low-resource conditions is very active (zhang † corresponding author figure 1 : an example of an intent classification input for a large language model that includes intent descriptions. the figure was adopted from @xcite . @xcite @xcite . @xcite proposed an in-context learning classification method using large language models to classify intents in unseen domains. they provided detailed intent descriptions as inputs to compensate for the lack of user query examples for each intent. figure 1 illustrates how descriptions are included in the in-context learning input. provided descriptions can capture the subtle semantic nuances and exceptions that are challenging to address with intent names alone. however, the paper does not clarify the quality or quantity of descriptions that should be used for training or inference, leaving practitioners without concrete guidelines. this paper aims to provide specific guidelines on the effective and efficient use of intent descriptions during training and testing for intent classification with large language models in unseen domains. this study specifically explores how to utilize intent descriptions in large language models through the following aspects: 1) effect of description quality: using chat-gpt (openai, 2023), the study collects intent descriptions for the clinc150 @xcite , hwu64 @xcite , and banking77 @xcite datasets. three sets of descriptions are collected: dependent descriptions, which consider semantic differences between intents, independent descriptions, generated without considering semantic differences, and cleansed descriptions manually filtered to address the subtle semantics. the impact of description quality on training and testing is investigated. 2) impact of description quantity: the study examines the effect of increasing the number of descriptions used for training on intent classification accuracy. 3) input length management: usage of off-the-shelf rankers, selecting the most probable intent options based on the similarity between the user query and descriptions, is examined to address the input length issue caused by descriptions. the optimal number of intent options to select that balances the trade-off between input length and performance is investigated. the study uses flan-t5 3b @xcite which is an instruction-tuned model of t5 3b model @xcite . our findings and contributions can be summarized as follows: ## conclusions this paper thoroughly explored the impact of intent description quality and quantity on zeroshot intent classification using large language models while addressing the challenges of increased input length. the results show that fine-tuned models with descriptions are more effective for intent classification with descriptions. additionally, higher-quality descriptions for both training and testing enhance performance, particularly during testing. using an off-the-shelf ranker to reduce input length by selecting the top ten intent options minimizes input length without significant trade-offs in performance. overall, this study provides practical guidelines for leveraging intent descriptions with large language models to address intent classification in low-resource settings.