Z ero DL : Zero-shot Distribution Learning for Text Clustering via Large Language Models (2025)

Abstract: The advancements in large language models (LLMs) have brought significant progress in NLP tasks. However, if a task cannot be fully described in prompts, the models could fail to carry out the task. In this paper, we propose a simple yet effective method to contextualize a task toward a LLM. The method utilizes (1) open-ended zero-shot inference from the entire dataset, (2) aggregate the inference results, and (3) finally incorporate the aggregated meta-information for the actual task. We show the effectiveness in text clustering tasks, empowering LLMs to perform text-to-text-based clustering and leading to improvements on several datasets. Furthermore, we explore the generated class labels for clustering, showing how the LLM understands the task through data.

## introduction large language models (llms) have demonstrated impressive performances on various downstream tasks @xcite . these also exhibit the ability to understand the context of input text, known as in-context learning (icl) @xcite . icl allows leveraging llms for specific tasks without further extensive training. however, effective use of icl hinges on well-designed prompts. while prompts with few-shot examples demonstrably improve performance, they can easily overfit a model to the examples @xcite ; @xcite ; inter alia). this led to a growing interest in zero-shot learning, which reduces the need for intricate few-shot selection. recent advancements in zero-shot learning involve incorporating more sophisticated use of prompt structures, such as chain-of-thought @xcite , zero-shot reasoning @xcite , and models trained to follow instructions @xcite . however, how to design prompts for target tasks remains challenging. while in-context learning (icl) relies on examples (d) tailored to specific tasks, zerodl aggregates all the outputs from these zero-shot inferences ( m), resulting in meta-level information ( m). this information is then used by the llm to generate its final predictions. motivated by the core principle of iclproviding task and data contexts within promptswe propose an approach to construct more effective zero-shot prompts by understanding how llms describe datasets across prompting outputs. as illustrated in figure 1 , zero-shot distribution learning (zerodl) aims to learn data distributions through zero-shot inferences. the method comprises two key components: open-ended zeroshot inference and output aggregation. zero-shot prompts are then constructed with the generated meta information, and used for actual task. this method takes advantage of the self-generated frame of llms to successfully carry out a given task. we exemplify the effectiveness of zerodl on text clustering tasks where complete task descriptions cannot be provided due to an absence of ground-truth class labels. in addition, our method works in a text-to-text format, allowing clustering with specific context. for instance, "i love this movie" and "i hate this movie" express opposite sentiment but belong to the same cluster of movie reviews (see our contributions in this paper are as follows: ## related works the majority of existing llm-based clustering approaches rely on traditional methods like k-means, employing llm-generated embeddings as input @xcite . clusterllm @xcite introduced llm-guided refinement of embedding-based clustering models. in contrast, our approach zerodl, directly leverages text-level prompting to inject specific viewpoints into llms, enabling more targeted and contextualized clustering. other works have explored using llms as clustering models @xcite @xcite , but these approaches typically relied on few-shot settings with access to gold labels. idas @xcite addressed this limitation by selecting representative data using an auxiliary embedding model to generate pseudolabels without demonstrations. these labels are subsequently refined through in-context learning. zerodl distinguishes itself by operating in a fully zero-shot setting, eliminating the need for additional embedding models. moreover, while idas's class labels are primarily influenced by a limited set of data points and generate the labels independently, zerodl considers the entire data distribution through an aggregation step. this enables our method to generate class labels in an auto-regressive manner, taking into account the relationships between different labels. the importance of appropriate ground-truth labels extends beyond clustering and permeates icl. while @xcite observed cases where the input-label correspondence does not play significant roles, @xcite argued that the impact heavily depends on target tasks and experiment settings. we believe that this work would serve as a reference that appropriate class labels can be successfully generated by llms themselves. ## results figure 2 illustrates the confusion matrix of the results. the relatively low performance of sst-5, even with gold labels, underscores the inherent difficulty of sentiment classification. in contrast, ag-news demonstrates successful matching with our method. this observation emphasizes the potential value of zerodl in scenarios where ground-truth labels are unavailable. data method classlabels sst-5 zerodl neutral sentiment: this class includes all the sentiment labels that express a neutral sentiment towards the movie or documentary. examples include (...) negative sentiment: this class includes all the sentiment labels that express a negative or sad emotion towards the movie or documentary. examples include (...) ambiguous sentiment: this class includes all the sentiment labels that do not clearly express a positive or negative emotion towards the movie or documentary. examples include (...) mixed sentiment: this class includes all the sentiment labels that express a mixed sentiment towards the movie or documentary. examples include (...) positive sentiment: this class includes all the sentiment labels that express a positive emotion towards the movie or documentary. examples include (...) idas disappointing, critical, negative, positive, disappointing or unsatisfying gold very negative, negative, neutral, positive, very positive agnews zerodl international relations and politics: this class includes topics related to international relations, diplomacy, middle east politics, terrorism, nuclear politics, and elections. sports and entertainment: this class includes topics related to sports, tennis, golf, basketball, baseball, cricket, and entertainment. business and economy: this class includes topics related to the economy, finance, stocks, mergers and acquisitions, retail, real estate, and labor markets. technology and science: this class includes topics related to technology, computing, internet, cybersecurity, space exploration, and science. idas iran's nuclear program, oil prices, businesses prioritize hardware upgrades in economic recovery, volcanic activity at mount st. helens gold world, sports, business, sci/tech the results demonstrate that clustering performance is generally higher when using manually selected labels than using the original dataset labels. this suggests that carefully chosen labels can significantly improve clustering outcomes. in the context, zerodl performs particularly well on sst-5, implying that the potential of zerodl to capitalize on informative class labels automatically. zerodl involves a trade-off in computational cost. we evaluate the impact of varying input data amounts on performance (see ## conclusion we introduce zerodl, a novel approach to contextualize tasks for a given llm. zerodl employs open-ended zero-shot inference and output aggregation to learn data distributions. we demonstrate its effectiveness, showing competitive performances against embedding-based clustering methods and superior performance than ground-truth labels in some cases. beyond its clustering capabilities, ze-rodl offers the generation of informative class labels that provide deeper insights into llms.