Synergizing Large Language Models and Pre-Trained Smaller Models for Conversational Intent Discovery (2024)

Abstract: In Conversational Intent Discovery (CID), Small Language Models (SLMs) struggle with overfitting to familiar intents and fail to label newly discovered ones. This issue stems from their limited grasp of semantic nuances and their intrinsically discriminative framework. Therefore, we propose Synergizing Large Language Models (LLMs) with pre-trained SLMs for CID (SynCID). It harnesses the profound semantic comprehension of LLMs alongside the operational agility of SLMs. By utilizing LLMs to refine both utterances and existing intent labels, SynCID significantly enhances the semantic depth, subsequently realigning these enriched descriptors within the SLMs’ feature space to correct cluster distortion and promote robust learning of representations. A key advantage is its capacity for the early identification of new intents, a critical aspect for deploying conversational agents successfully. Additionally, SynCID leverages the in-context learning strengths of LLMs to generate labels for new intents. Thorough evaluations across a wide array of datasets have demonstrated its superior performance over traditional CID methods.

## introduction recognizing user intents within conversational utterances is pivotal for developing intelligent conversational agents @xcite @xcite . previous research mainly formulates this problem as a close-world intent classification task @xcite . however, in real-world applications, new intents continuously emerge. this spurs increasing interest in the open-world conversational intent discovery (cid) @xcite @xcite , a task that aims to recognize both known and new intents from extensive or even limited amount of user utterances @xcite . current attempts at cid primarily rely on pretrained small language models (slms), which fall into two main categories: unsupervised and semisupervised. unsupervised methods @xcite firstly train slms without using any labeled data to obtain utterance representations, and then cluster them to infer intents. in contrast, semi-supervised methods @xcite @xcite leverage the available labeled data for the initial pre-training of slms, followed by fine-tuning these models with pseudo supervisory signals on unlabeled utterances for intent recognition. given the specialized agility of slms, these methods can easily fit user utterances and learn discriminative representations for cid. however, two key challenges persist. the first is overfitting to known intents, where these methods struggle to capture the full scope of intents and accurately model known label semantics. this limitation not only biases them towards existing intent categories but also compromises their ability to detect new intents early, a crucial capability for adaptive conversational agents. the second challenge is the inability to label novel intents, due to the inherently discriminative architecture of cid models, which falls short in recognizing and labeling emerging intents, marking a critical adaptability gap in current approaches. recently, large language models (llms) @xcite @xcite have achieved significant breakthroughs in language understanding and generative tasks, including summarization @xcite and query rewriting @xcite . their success inspires a potential solution for addressing the above challenges by adapting llms to enhance intent discovery. yet, the context length limitation of llms restricts their direct use in cid, which requires clustering thousands of utterances. while integrating user utterances with task-specific prompts to solicit intent labels from llms is possible, this prompting method risks generating intent labels without sufficient control, thus leading to unpredictable and uninstructive outcomes @xcite . to navigate these challenges while leveraging the strengths of both llms and slms, we introduce syncid, a framework that synergizes the deep semantic insights of llms with the agile, specialized capabilities of slms. syncid employs a dual-prompting mechanism with llms to refine both utterances and known intent labels, enhancing the semantic precision of intent descriptors. this refinement process, informed by the nuanced understanding of llms, not only clarifies the intent representation but also primes the data for more effective learning. following this, slms are trained through contrastive learning to align the semantic spaces of utterances with those of the intent descriptors. this innovative alignment strategy significantly reduces cluster distortion and improves the system's ability to detect and label new intents early, addressing the primary limitations of current cid approaches. by selecting a limited number of close-to-center utterances from newly formed intent clusters for in-context learning with llms, syncid achieves precise intent labeling. the main contributions of this work can be summarized as follows: ## conclusion in this paper, we introduced syncid, a novel framework that can effectively synergize llms and pretrained slms for conversational intent discovery. by aligning llms' reliable insights with the agile responsiveness of specialized slms, syncid effectively alleviates the risk of overfitting to known intents in cid. furthermore, syncid enables the llms with in-context learning to skillfully produce labels for newly identified intent clusters. through extensive experiments, our findings confirm syn-cid's effectiveness. deeper analysis reveals that syncid not only sets new benchmarks in cid but also generates appropriate intent labels and enables early detection of new intents. limitations despite the promising results obtained by our syn-cid, it is important to acknowledge several limitations: (1) the syncid's reliance on llms subjects it to llms' inherent flaws, including biases in the training data and the propensity for hallucinating incorrect information. (2) the financial cost of utilizing commercial llm apis, such as openai's, for experiments is significant. in our case, accessing apis of llms such as gpt-4, gpt-3.5-turbo, and davinci-003 for getting all the experimental results incurred a cost of approximately $510. (3) our syncid, similar to existing baselines, assumes a known ground-truth number of intents for clustering utterances -a condition that diverges from real-world applications where the exact number of intents remains unknown. to validate syncid's effectiveness and robustness, we conduct further experiments with an estimated number of intents and explore the impact of various intent numbers around it on the cid performance of our syncid. the findings from these additional experiments are detailed in appendix b.