SPILL : Domain-Adaptive Intent Clustering based on Selection and Pooling with Large Language Models (2025)

Abstract: In this paper, we propose Selection and Pooling with Large Language Models (SPILL), an intuitive, domain-adaptive method for intent clustering without fine-tuning. Existing embeddings-based clustering methods rely on a few labeled examples or unsupervised fine-tuning to optimize results for each new dataset, which makes them less generalizable to multiple datasets. Our goal is to make these existing embedders more generalizable to new domain datasets without further fine-tuning. Inspired by our theoretical derivation and simulation results on the effectiveness of sampling and pooling techniques, we view the clustering task as a small-scale selection problem. A good solution to this problem is associated with better clustering performance. Accordingly, we propose a two-stage approach: First, for each utterance (referred to as the seed), we derive its embedding using an existing embedder. Then, we apply a distance metric to select a pool of candidates close to the seed. Because the embedder is not optimized for new datasets, in the second stage, we use an LLM to further select utterances from these candidates that share the same intent as the seed. Finally, we pool these selected candidates with the seed to derive a refined embedding for the seed. We found that our method generally outperforms directly using an embedder, and it achieves comparable results to other state-of-the-art studies, even those that use much larger models and require fine-tuning, showing its strength and efficiency. Our results indicate that our method enables existing embedders to be further improved without additional fine-tuning, making them more adaptable to new domain datasets. Additionally, viewing the clustering task as a small-scale selection problem gives the potential of using LLMs to customize clustering tasks according to the user’s goals.

## introduction intent detection is a fundamental component in task-oriented dialogue (tod) systems, aimed at classifying user utterances into pre-defined intent categories @xcite . although some research has focused on addressing data scarcity @xcite , pre-defined intent labels are insufficient for addressing all user requests, as new intents emerge with growing complexity of customer requirements and appearance of novel domains on the business front. while the progress of transformer-based models has greatly enhanced intent detection performance, the identification of emerging intents in task-based conversational agents continues to present a challenge @xcite . to address this issue, the majority of research aims to develop embedding models that group unlabeled utterances into clusters based on a labeled or unlabeled in-domain dataset @xcite @xcite . the goal of these approaches is to enable the embedder to learn a robust representation of user utterances while aligning with the cluster objective. contrastive learning is commonly employed for this purpose, aiming to learn a representation through comparison @xcite . the clustering objective is achieved through the design of a cluster loss function @xcite @xcite . although these approaches yield good results, they require finetuning for each dataset. @xcite . in recent years, advancements in generative large language models (llms) @xcite have facilitated improvements in intent clustering. @xcite and @xcite used llms to guide the fine-tuning of embedders, aiming to align the embeddings' clustering outcomes with llms' interpretations. although these studies achieve state-of-the-art results, they face two primary challenges: first, modifying the loss function adds complexity, as it involves designing different loss functions and tuning additional hyperparameters, such as the weight of each loss term. this makes optimization more difficult. second, building a new embedder requires optimization for each dataset, which limits generalizability. in this paper, we propose a theoretical framework for clustering, grounded by formal proofs, and an intuitive and effective approach to address these challenges. our approach has three key goals: easy implementation, no need for fine-tuning, and the ability to adapt to unseen datasets. our approach stays close to a theoretical rationale and we confirm its potential by simulation analysis. the key idea is that if we can identify a few utterances that share the same cluster as the seed utterance from a randomly selected subset, pooling the seed utterance with these selected utterances will bring them closer to the cluster centroid. based on this premise, we can see a clustering task as a smallscale selection problems. our approach consists of two stages: in the first stage, for each utterance (referred to as the "seed utterance"), we use an existing embedder (a traditional encoder or a decoder-only llm) to gain a larger pool of similar utterances. in the second stage, we use llms to further select utterances that share the same intent cluster as the seed utterance. note that our proposed approach is not intended to compete with other embedders. instead, it is designed to complement most existing approaches and can strengthen each of the existing models. with our experiments, we show that our method can boost performance on the clustering task irrespective of the used embedder. in summary, we make three contributions: (1) we provide a theoretical framework supported by formal proofs and simulations, which frame the clustering task as a smallscale selection problem, providing both theoretical and empirical contributions to the task. (2) we propose a novel and easy-to-implement approach that is generalizable, regardless of the embedders used and does not require fine-tuning and can operate with low computational resources; (3) we show our method enables domain adaptation in clustering for unseen datasets, achieving state-of-the-art results on four benchmark collections. ## related work intent clustering with contrastive learning. grouping user utterances and identifying new intents is essential in tod systems @xcite @xcite @xcite . most research has focused on developing embedding models to create strong representations of user utterances. for instance, @xcite pretrained a model with little labelled data and use k-means to produce cluster assignments as pseudo labels. they learn the intent representations under the supervision of the aligned pseudo-labels. @xcite propose a method that optimizes both the contrastive loss and clustering loss together to build a sentence embedding model. to prevent overfitting on in-domain data during contrastive loss optimization, @xcite limit the comparison to k-nearest neighbors instead of considering all possible neighbors. as earlier research focused on contrastive learning without fully accounting for the semantic meanings of labels, @xcite use two-level contrastive learning to learn representations. this approach first aligns embeddings with several contrastive objectives, including their proposed label semantic alignment, then applies soft prompting to enhance the use of semantic knowledge in intent discrimination. ## theoretical framework in this section, we introduce our theoretical derivation, followed by validations through empirical simulations. ## computational method ## conclusions we proposed selection and pooling with large language models (spill), an intuitive and domainadaptive method for intent clustering without finetuning. spill is applicable to any embedder and does not require fine-tuning. we found that: (1) our proposed method enables viewing a clustering task as a small-scale selection problem, providing a novel perspective on the clustering process; (2) our method demonstrates general improvements regardless of the embedding model used, highlighting its versatility; (3) our method can achieve comparable results to other sota research studies without finetuning and by using smaller models; (4) it proves to be effective in data-limited domains, showing its adaptability in challenging scenarios. for future work, we see potential for improved prompting, the addition of few-shot settings, and generalisation to other languages.