C luster LLM : Large Language Models as a Guide for Text Clustering (2023)

Abstract: We introduce ClusterLLM, a novel text clustering framework that leverages feedback from an instruction-tuned large language model, such as ChatGPT. Compared with traditional unsupervised methods that builds upon “small” embedders, ClusterLLM exhibits two intriguing advantages: (1) it enjoys the emergent capability of LLM even if its embeddings are inaccessible; and (2) it understands the user’s preference on clustering through textual instruction and/or a few annotated data. First, we prompt ChatGPT for insights on clustering perspective by constructing hard triplet questions < does A better correspond to B than C > , where A, B and C are similar data points that belong to different clusters according to small embedder. We empirically show that this strategy is both effective for fine-tuning small embedder and cost-efficient to query ChatGPT. Second, we prompt ChatGPT for helps on clustering granularity by carefully designed pairwise questions < do A and B belong to the same category > , and tune the granularity from cluster hierarchies that is the most consistent with the ChatGPT answers. Extensive experiments on 14 datasets show that ClusterLLM consistently improves clustering quality, at an average cost of ~$0.6 per dataset.

## preliminary text clustering takes an unlabeled corpus @xmath0, and outputs a clustering assignment @xmath1. to specify user's needs, cluster-llm integrates additional textual instruction (e.g. "select the example that better corresponds with the query in terms of entity type.") to understand perspective and few-shot annotations (e.g. "sentence1 and sentence2 have the same entity type ...") to determine cluster granularity. ## our clusterllm clusterllm is based on a pre-trained small embedder @xcite (denoted as f ) which usually represents sentences individually. in contrast, inspired by human cognitive ability (nosofsky, 2011), clusterllm considers a pair or a triplet of sentences through prompting llms that are trained to follow human instructions @xcite . specifically, clusterllm is a two-stage framework (see figure 2 ). in section 3.1 we introduce stage 1 that utilizes triplet task to improve clustering quality with respect to user-specified perspectives, along with a sampling strategy that reduces number of api queries. in section 3.2, we introduce stage 2 that leverages pairwise task to determine cluster granularity based on predictions from llms. ## experiments we first evaluate clusterllm on clustering quality with ground truth number of clusters in section 4.4. then we conduct ablation studies in section 4.6 to further analyze the effectiveness of clusterllm. finally, we show results of determining cluster granularity in section 4.7. ## conclusion in this paper, we study how to leverage api-based llms to guide small embedders for text clustering in order to benefit from high-level language capability of llms and user's instructions on clustering. we propose to prompt llms with two kinds of sentence relationship tasks: triplet task and pairwise task. triplet task chooses the sentence that is most similar with anchor combining with a perspective instruction from users. the predicted triplets are used for fine-tuning small embedders. pairwise task judges whether a pair of sentences belong to the same category hinted by few-shot demonstrations, and then the predictions are used to determine cluster granularity with a consistency measure. extensive experiments show that our proposed framework clusterllm can improve clustering quality and propose reasonable cluster granularity at a negligible cost. however, clus-terllm still relies on the embedding model itself, which is inefficient and inapplicable on black-box embedding models. we encourage future works to explore the potential of model-free training such as constrained clustering.