Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of C hat GPT (2023)

Abstract: The tasks of out-of-domain (OOD) intent discovery and generalized intent discovery (GID) aim to extend a closed intent classifier to open-world intent sets, which is crucial to task-oriented dialogue (TOD) systems. Previous methods address them by fine-tuning discriminative models. Recently, although some studies has been exploring the application of large language models (LLMs) represented by ChatGPT to various downstream tasks, it is still unclear for the ability of ChatGPT to discover and incrementally extent OOD intents. In this paper, we comprehensively evaluate ChatGPT on OOD intent discovery and GID, and then outline the strengths and weaknesses of ChatGPT. Overall, ChatGPT exhibits consistent advantages under zero-shot settings, but is still at a disadvantage compared to fine-tuned models. More deeply, through a series of analytical experiments, we summarize and discuss the challenges faced by LLMs including clustering, domain-specific understanding, and cross-domain in-context learning scenarios. Finally, we provide empirical guidance for future directions to address these challenges.

## introduction traditional task-oriented dialogue (tod) systems are based on the closed-set hypothesis @xcite @xcite and can only handle queries within a limited scope of in-domain (ind) intents. however, users may input queries with out-of-domain (ood) intents in the real open world, which poses new challenges for tod systems. recently, a series of tasks targeting ood queries have received extensive research. ood intent discovery @xcite @xcite aims to group ood queries into different clusters based on their intents, which facilitates identifying potential directions and developing new skills. the general intent discovery (gid) task @xcite further considers the increment of ood intents and aims to automatically discover and increment the intent classifier, thereby extending the scope of recognition from existing ind intent set to the open world, as shown in fig 1 . previous work studied above ood tasks by finetuning the discriminative pre-training model bert @xcite . recently, a series of powerful generative llms have been proposed one after another, such as gpt-3 @xcite , palm @xcite and llama @xcite . the emergence of llms has brought revolutionary changes to the field of natural language processing (nlp). given the superior in-context learning ability, prompting llms has become a widely adopted paradigm for nlp research and applications @xcite . since these llms are trained on a large amount of general text corpus and have excellent generalization ability, this triggers the thinking of what benefits llms can bring and what challenges will llms face when applying them to open-scenario intent discovery and recognition? as one of the representative llms, chatgpt, developed by openai, has attracted significant attention from researchers and practitioners in a short period of time. while the nlp community has been studying the ability of llms to be applied to various downstream tasks, such as translation @xcite , mathematics @xcite , education @xcite ,emotion recognition @xcite , its ability in ood is still not fully explored. different from @xcite which evaluate the robustness of chatgpt from the adversarial and out-of-distribution perspective, ood intent discovery and gid focuses on using ind knowledge transfer to help improve tod systems with ood data. in this paper, on the one hand, we focus on ood intent discovery and gid the two open-scenario intent tasks to explore whether prompting chatgpt can achieve good performance in discovering and incrementally recognizing ood intents; on the other hand, we aim to gain insights into the challenges faced by llms in handling open-domain tasks and potential directions for improvement. to the best of our knowledge, we are the first to comprehensively evaluate chatgpt's performance on ood intent discovery and gid. in detail, we first design three prompt-based methods based on different ind prior to guide chatgpt to perform ood discovery in an end-to-end manner. for gid, we innovatively propose a pipeline framework for performing gid task under a generative llm (section 3). then we conduct detailed comparative experiments between chatgpt and representative baselines under three dataset partitions (section 4). in order to further explore the underlying reasons behind the experiments, we conduct a series of analytical experiments, including in-context learning under cross-domain demonstrations, recall analysis, and factors that affect the performance of chatgpt on ood discovery and gid. finally, we compare the performance of different llms on these ood tasks (section 5). our findings. the major findings of the study include: what chatgpt does well: ## challenge & future work based on above experiments and analysis, we summarize three challenging scenarios faced by llms and provide guidance for the future. ## conclusion in this paper, we conduct a comprehensive evaluation of chatgpt on ood intent discovery and gid, and summarize the pros and cons of chat-gpt in these two tasks. although chatgpt has made significant improvements in zero or few-shot performance, our experiments show that chatgpt still lags behind fine-tuned models. in addition, we perform extensive analysis experiments to deeply explore three challenging scenarios faced by llms: large-scale clustering, domain-specific understanding, cross-domain in-context learning and provide guidance for future directions.