Navigating the Unknown: Intent Classification and Out-of-Distribution Detection Using Large Language Models (2025)

Abstract: Out-of-Distribution (OOD) detection is a challenging task that requires great generalization capability for the practicality and safety of task-oriented dialogue systems (TODS). With the dawn of large language models (LLMs), their enhanced ability to handle diverse patterns and contexts may aid in addressing this challenging task. In this paper, we investigate the current performance of LLMs in the near-OOD setting, where OOD queries belong to the same domain but different intents. To take advantage of out-of-the-shelf capabilities of LLMs, we do not use fine-tuning. We study the performance of one of the leading frontier models, GPT-4o, in 3 well-known public datasets and 3 in-house datasets, using 10 different methods and prompt variations. We study the performance of different prompts and techniques in Gemini 1.5 Flash and Llama 3.1-70b. We investigate the effect of increasing the number of In-Distribution (ID) intents. We propose a novel hybrid method that is cost-efficient, high-performing, highly robust, and versatile enough to be used with smaller LLMs without sacrificing performance. This is achieved by combining ID success of smaller text classification models and high generalization capabilities of LLMs in OOD detection.

## introduction as the field of natural language processing (nlp) progresses rapidly, task-oriented dialogue systems (tods) are experiencing a significant increase in their overall capabilities. their efficiency, accessibility, and coverage have improved with the emergence of the large language model (llm) paradigm, as shown by @xcite . in tods, natural language understanding (nlu) tasks begin with intent detection, where the user query is mapped to a set of known intents to control the flow of the dialogue, select appropriate knowledge sources, and so on. prior to llms, this task was handled by transformer models such as bert @xcite and roberta (liu et al., 2019). these models operate under the closed world hypothesis, as discussed by @xcite , and can only detect what is present in their training data. however, the real world is open-ended, and users often submit unseen or unrelated queries. the need to reject such out-of-scope queries -illustrated in figure 1 -has led to the development of out-of-distribution (ood) detection systems, described in surveys by @xcite and @xcite . the improved generalization capabilities introduced by llms hold great promise for addressing this challenge. as an intent detection model is trained with a fixed set of intents and utterances, the problem of ood detection can be framed as the task of identifying a distribution shift. in nlp, two primary types of distributional shift are typically discussed. the first is semantic shift, where ood queries arise from a different intent space and must be filtered before being incorrectly mapped to known intents. the second is covariate shift, where the intent space remains the same, but the input distribution changes, leading to novel utterances for familiar intents, as discussed by @xcite and @xcite . in this paper, we focus on semantic shifts. the challenges of ood detection extend beyond distributional shifts. training datasets, often constructed by domain experts without machine learning expertise, may be imbalanced, with certain intents represented by very few examples. additionally, the scope of intents can vary widely -some may be fine-grained while others are overly broad. this work focuses on a more demanding subset of the problem, known as near-ood detection @xcite , also referred to as in-domain and out-of-scope detection (id-oos) by @xcite . in this setting, unknown examples do not come from entirely different domains but instead belong to different intent labels within the same general distribution. near-ood detection mimics the presence of long-tail intents unseen during training by identifying semantically similar yet label-divergent in-domain examples. in this paper, we investigate the performance of llms on near-ood detection without fine-tuning. we aim to provide insights into the latest models, their performance with different techniques, and how they can be better utilized. our main contributions are: 1. we investigate the performance of gpt-4o in the challenging task of near-ood detection using 3 well-known public datasets, and 3 in-house datasets. we use zero-shot and few-shot prompts and their k-nearest (k-n) variations, as well as a hybrid method with various prompting strategies. the results demonstrate exceptional performance, surpassing prior studies on the benchmarks. 2. we introduce a novel hybrid method. our method, on average, is the highest performing amongst all strategies. it also improves the performance of smaller or open source llms to match gpt4o, reduces the input-token numbers and cost, is robust, and easy to implement. ## experiments and results we devise three different experiments. in sec 4.1, we want to see how different approaches compare to each other. to that end, we employ all the methods in sec 3.3 using gpt-4o as the llm representative. in sec 4.2, we aim to compare the performance of different llms using zero-shot and few-shot prompting with all the intents, and using the hybrid method with balanced prompt and contrastive prompt. for this purpose, we use gpt-4o (openai, 2024a), gemini 1.5 flash (google, 2024), and llama 3.1-70b-instruct @xcite . lastly, in sec 4.3, the goal is to see the effect of the number of id intents on ood detection performance. we use the same llms and methods as in the previous experiment. ## discussion ood detection is a challenging task that is critical for the practicality and safety of ai systems. our study sheds light on the current state of one of the frontier models, gpt-4o, which performs significantly better than its older successors like gpt-4 and gpt-3.5, as reported by @xcite . their experiments on banking77 using gpt 3.5 with zero-shot prompting show 33.8 ood f1, whereas our gpt-4o experiments score 71.6 with the same combination, as can be seen in in few-shot prompting, @xcite reports the performance of gpt 3.5 with different numbers of intents and different numbers of utterances per intent. they demonstrate a significant performance drop in ood f1. from 5 intents to 40 intents, gpt 3.5 suffers a loss of approximately ∼50 points. in we also believe the performance of ood detection strongly depends on the dataset quality. if any intent has a misleading label name, has noisy examples, or it has utterance examples that can be used in another intent as well, the results get affected significantly. we suggest that the low performance in banking77 in comparison to other public datasets is due to these kinds of effects, as reported by @xcite . in bit3 datasets, the results in ## conclusion and future work in this paper, we extensively study the current state of llms in one of the challenging tasks of nlu, the near-ood detection task. using 10 methods across 3 public and 3 in-house datasets, we evaluate gpt-4o and compare it with gemini 1.5 flash and llama 3.1-70b. we study how the increasing number of intents affects the performance of gpt-4o. we introduce a novel hybrid method that is robust, high-performing, easy to use, that enables the usage of smaller or open-source models without sacrificing performance. despite broad coverage, several research directions remain: cost and latency. by combining cost and latency into a metric we may find the most beneficial strategy to run large scale intent detection with ood detection support. alternative hybrid methods. the hybrid method provided here focuses on combining the id strength of transformer models with the generalization capability of llms. it is cost-efficient as it doesn't require every intent and utterance to be fed to llms; however, we still send each intent even though the baseline method is extremely confident. using a post hoc method and using llms only when the method is unsure may provide further efficiency and speed. detailed llm comparison. to clearly see the comparison of different providers and open-source models, we need thorough experimentation. the frontier models, the newly emerging reasoning models, and small language models are all rich future directions. multi-label intent detection. intent detection systems often do a simplification that is rather unrealistic of human communication. it is to expect each utterance to have a single intent. in reality, communication often involves more intricate information having multiple levels of intentions. one may ask a couple of things at once, or ask a more abstract concept with non-trivial bounds, or ask something that can be understood in different levels of detail. the llms open the door for such advanced nlu cases.