Generalized Category Discovery with Large Language Models in the Loop (2024)

Abstract: Generalized Category Discovery (GCD) is a crucial task that aims to recognize both known and novel categories from a set of unlabeled data by utilizing a few labeled data with only known categories. Due to the lack of supervision and category information, current methods usually perform poorly on novel categories and struggle to reveal semantic meanings of the discovered clusters, which limits their applications in the real world. To mitigate the above issues, we propose Loop, an end-to-end active-learning framework that introduces Large Language Models (LLMs) into the training loop, which can boost model performance and generate category names without relying on any human efforts. Specifically, we first propose Local Inconsistent Sampling (LIS) to select samples that have a higher probability of falling to wrong clusters, based on neighborhood prediction consistency and entropy of cluster assignment probabilities. Then we propose a Scalable Query strategy to allow LLMs to choose true neighbors of the selected samples from multiple candidate samples. Based on the feedback from LLMs, we perform Refined Neighborhood Contrastive Learning (RNCL) to pull samples and their neighbors closer to learn clustering-friendly representations. Finally, we select representative samples from clusters corresponding to novel categories to allow LLMs to generate category names for them. Extensive experiments on three benchmark datasets show that Loop outperforms SOTA models by a large margin and generates accurate category names for the discovered clusters. Code and data are available at https://github.com/Lackel/LOOP.

## introduction although modern machine learning systems have achieved superior performance on many tasks, the vast majority of them follow the closed-world setting that assumes training and test data are from the same set of pre-defined categories @xcite . however, in the real world, many practical problems such as intent detection @xcite and image recognition @xcite are open-world, where the well-trained models may encounter data with unseen novel categories. to cope with this limitation, generalized category discovery (gcd) was proposed and widely studied in both nlp @xcite and computer vision @xcite . gcd requires models to recognize both known and novel categories from a set of unlabeled data, based on some labeled data containing only known categories, which can adapt models to the emerging categories without any human efforts. current methods @xcite @xcite @xcite usually first perform supervised pretraining on labeled data and self-supervised learning on unlabeled data to train a base model such as bert @xcite , then they perform clustering methods such as kmeans to discover both known and novel categories. even though these meth-ods can improve performance on known categories, they usually perform poorly on novel categories due to the lack of supervision. furthermore, they also struggle to reveal semantic meanings (e.g., category names or descriptions) of the discovered clusters due to the lack of prior knowledge for novel categories. recently, large language models (llms) such as chatgpt have shown extraordinary capabilities for various applications even without any labeled samples @xcite . however, llms cannot be directly applied to gcd which requires models to cluster thousands of samples. and problems such as data privacy, high inference latency, and high api cost also limit their applications in the real world. to solve the above limitations and enjoy the benefits of both base models and llms, we propose loop, an end-to-end active-learning framework that introduces llms into the training process. by selecting a few key samples to query llms and optimizing the base model based on the feedback, loop can compensate for the lack of supervision and generate category names for the discovered clusters with little query cost. hence, we only need to train and maintain a small base model locally, which can reduce the inference cost and protect data privacy. specifically, as shown in fig. 1 , we first propose local inconsistent sampling (lis) to select the most informative samples that have a higher probability of falling into wrong clusters. specifically, we select samples that have high entropy of cluster assignment probabilities and whose neighbors have the most diverse cluster assignments. intuitively, samples with high entropy and diverse neighbor predictions seem to violate the clustering assumption @xcite and locate near decision boundaries (fig. 2 dashed circle), so these neighbor-chaotic samples with great uncertainty would have a high probability of falling to wrong clusters @xcite , so correcting them can significantly improve the model performance. after selecting the key samples, we need to build proper prompts to query the llms. however, due to the lack of information for novel categories, we cannot directly query llms which category these samples belong to as in traditional active learning. to solve this issue, we propose a scalable query strategy that allows llms to choose true neighbors of the selected samples from multiple candidate neighbor samples @xcite . based on the feedback of llms, we can solve the local inconsistency problem and decide which clusters these key samples truly belong to. furthermore, llms are more competent at comparing semantic similarities between sentences than choosing from multiple category names. then based on the refined neighbor relationships, we perform refined neighborhood contrastive learning (rncl) to pull samples closer to their neighbors to learn clustering-friendly representations. in this way, we can correct these samples by pulling them closer to the true clusters they belong to and cluster the rest of the samples to form more compact clusters. finally, we decouple the clusters corresponding to novel categories from the discovered clusters @xcite and select a few samples closest to each center of the clusters to query llms to generate category names for novel categories. experimental results on three benchmark datasets show that loop outperforms sota models by a large margin and generates accurate category names for the discovered clusters. furthermore, we also validate that the proposed local inconsistent sampling strategy can select more informative samples and the proposed scalable query strategy can help to correct the selected samples effectively with minimal query cost. our contributions can be summarized as follows: labeled data encoder l rncl encoder sports news science ⅰ: model training ⅱ: cluster interpretation cluster name known novel unlabeled data which one is more like is more like or ? query llms it is 'sports'.