Intent Recognition and Out-of-Scope Detection using LLM s in Multi-party Conversations (2025)

Abstract: Intent recognition is a fundamental component in task-oriented dialogue systems (TODS). Determining user intents and detecting whether an intent is Out-of-Scope (OOS) is crucial for TODS to provide reliable responses. However, traditional TODS require large amount of annotated data. In this work we propose a hybrid approach to combine BERT and LLMs in zero and few-shot scenarios to recognize intents and detect OOS utterances. Our approach leverages LLMs generalization power and BERT’s computational efficiency in such scenarios. We evaluate our method on multi-party conversation corpora and observe that sharing information from BERT outputs to LLMs lead to system performance improvement.

## introduction advances in dialogue systems have facilitated their employment to assist users on daily tasks in domains such as banking, health consulting, hospitality and others @xcite @xcite . taskoriented dialogue systems (tods) in real-world applications must be able to both recognize user intents and detect out-of-scope (oos) intents to generate reliable responses. standard methods for intent recognition generally require large amounts of annotated data. however, annotations are scarce in some real-world applications, especially when new intents are introduced into systems. large language models have been shown to be robust at classification tasks in zero and few-shot settings. nevertheless, inferences from llms are computationally costly, thus their extensive use remains unpractical in some scenarios. previous work have proposed hybrid approaches combining llms and smaller language models, by only routing uncertain inferences to llms at inference time @xcite . these approaches consist in processing queries in two steps: first through the computationally efficient model, and then through llms, if necessary. in doing so, overall computational costs are reduced without compromising prediction quality. however, these methods do not share information among models, and hence llms miss potential relevant information from the preceding step. in this work we propose a hybrid approach that combines small language models and llms for intent recognition and oos detection in multi-party conversations, i.e. dialogues between three or more participants. we route inferences with high uncertainty from fine-tuned bert models to llms, and use the information from the outputs of the finetuned models to dynamically generate the prompts at inference time. such information is employed to reduce the label space on the classification task. experiments in this study are conducted on three open source llms. our work leverages the efficiency of (relatively) small models and the power of llms in zero-shot settings for intent classification and oos detection. figure 1 illustrates our proposed method. ## related work in recent years, intent detection methods have mainly consisted in fine-tuning small models @xcite @xcite . @xcite studied the use of class names to improve in-scope (is) intent classification and oos detection, using bert and spherical variational autoencoders @xcite . @xcite observed that fine-tuning sentence transformers presents largely better is and oos performance than traditional methods. llms have gained attention in multiple nlp tasks, including intent recognition in dialogue systems @xcite @xcite . findings are contradictory, as some works have found that llms outperform fine-tuned models @xcite and others have shown the opposite @xcite . to the best of our knowledge, the only study on intent recognition and oos detection with a focus on efficiently using llms is proposed in @xcite . they propose a hybrid method that uses sentence transformers and llms, which reduced the performance gap to 2% while reducing computing latency up to 50%. however, their approach does not consider sharing information among models. furthermore, we focus our work in multi-party conversations as such scenarios have been overlooked in previous work across most dialogue system tasks @xcite . ## experimental procedure let @xmath0, y i ) | y @xmath1, and y @xmath2, . . . , m, m + 1} denotes the set of m in-scope intents plus the out-of-scope label. our aim is to build a multiclass classification system that detects whether u i corresponds to an oos intent from an unknown distribution or whether u i can be classified into any of m possible in-scope intents. ## results ## computational efficiency analysis ## conclusions we investigated how (relatively) small language models such as bert can be combined with llms in zero-shot scenarios to reduce computational costs on intent recognition tasks without compromising predictive quality. our results on mpcs are in line with previous works in dyadic dialogues, suggesting that uncertainty-based routing lead to performance gains. our work also demonstrates that sharing information among models such as probability estimates to reduce the label space out-performs methods without shared information. future work may consider exploring other plausible label selection strategies. additionally, other sources of information to be leveraged in llm prompts from small models (i.e. bert) may be investigated in future studies: the actual probability estimates, uncertainty patterns, model's internal representations, etc. finally, although our experiments are conducted on multi-party corpora, our proposed method could also be applied on dyadic scenarios. we believe that our findings show promising directions towards robust and efficient intent recognition systems in real-world applications.