Topic Modeling for Short Texts with Large Language Models (2024)

Abstract: As conventional topic models rely on word co-occurrence to infer latent topics, topic modeling for short texts has been a long-standing challenge. Large Language Models (LLMs) can potentially overcome this challenge by contextually learning the meanings of words via pretraining. In this paper, we study two approaches to using LLMs for topic modeling: parallel prompting and sequential prompting. Input length limitations prevent LLMs from processing many texts at once. However, an arbitrary number of texts can be handled by LLMs by splitting the texts into smaller subsets and processing them in parallel or sequentially. Our experimental results demonstrate that our methods can identify more coherent topics than existing ones while maintaining the diversity of the induced topics. Furthermore, we found that the inferred topics cover the input texts to some extent, while hallucinated topics are hardly generated.

## introduction topic modeling is the classical task of discovering latent topics that best describe a set of documents @xcite . recently, while neural topic models have worked successfully on various kinds of long documents @xcite @xcite , they have not been able to handle short texts, such as social media posts and news headlines @xcite . large language models (llms), such as in-structgpt @xcite and @xcite , have shown impressive results on various tasks by providing task instructions in a zeroshot manner @xcite . since conventional topic models infer the topics of words by relying on word co-occurrence, they perform worse on short texts. in contrast, as llms contextually learn the meanings of words by pretraining on massive text corpora, they could accurately infer the latent topics. we propose two approaches to using llms for topic modeling: parallel prompting and sequential prompting (figure 1 ). due to the input length limitations of llms, an input document set must be split into smaller subsets, which are processed individually. parallel prompting concurrently infers the topics of each subset and merges them to represent the topics of the whole document set. sequential prompting processes each subset successively, updating the topics in every iteration. we assess our approaches across texts from various domains using multiple evaluation metrics. the contributions of this study are as follows: ## background topic modeling is the task of identifying latent topics as a set of topic words representing each topic from a collection of documents @xcite . topic modeling has conventionally been tackled with probabilistic models such as latent dirichlet allocation @xcite . in recent years, however, neural models have come into widespread use due to their high performance @xcite @xcite . it is known that topic modeling for short texts is difficult for current topic models due to data sparsity @xcite . tsctm @xcite is a current state-of-the-art neural topic model for short texts. this model addresses data sparsity by learning representations of documents using vq-vae @xcite , contrastive learning, and incorporation of data augmentation into the learning. bertopic @xcite uses a pretrained encoder, sentence-bert @xcite , to obtain clusters of documents and assigns topic words to each cluster by using a class-based tf-idf procedure. another related study is @xcite , in which llms are utilized to automatically evaluate topic quality. however, our study is the first to explore how well llms perform topic modeling. ## topic modeling with llms we introduce two approaches to performing topic modeling with llms: parallel prompting and sequential prompting. for these approaches, we apply common preprocessing, which involves randomly splitting a document set into subsets with the same size, smaller than the context length of the llms. ## experiments we investigate how well our methods perform topic modeling for short texts. ## results and discussion topic quality document coverage factuality as shown in parallel and sequential prompting to analyze the superior performance of the parallel approach, we calculated cv and dc of topics before merging. model # topics bertopic #1 kanye black thanksgiving west xbox #2 china independence zone scotland air #3 hiv aarushi watkins ian woman #4 jellyfish robot seahorse flying methane #5 alzheimer brain infant risk gene gpt-4par #1 kanye west kim kardashian parody #2 thanksgiving black friday shopping deal #3 xbox microsoft game console sale #4 nokia lumia microsoft smartphone tablet #5 syria peace talk geneva conference gpt-4seq #1 kanye west kim kardashian parody #2 black friday shopping thanksgiving deal #3 xbox game console playstation microsoft #4 comet ison sun spectacular encounter #5 scottish independence salmond white paper ## conclusion in this study, we proposed two approaches to using llms for topic modeling: parallel prompting and sequential prompting. we implemented our methods on gpt-3.5 and gpt-4 and evaluated their performance on three datasets together with three existing topic models. in the evaluation, in addition to the well-known metrics for topic quality, we introduced two new metrics, document coverage and factuality, to assess the potential issues with llms reflecting only some documents or outputting hallucinated topics. the results showed that llms could find higher-quality topics than existing methods, and the impact of these issues was not 10 examples and further analysis are provided in appendix c.3. remarkable in practice. future work will include improving our methods to enable topic assignment to each document. in natural language processing, pages 2748-2760, abu dhabi, united arab emirates. association for computational linguistics. jianhua yin and jianyong wang. 2016. a model-based approach for text clustering with outlier detection. in 2016 ieee 32nd international conference on data engineering (icde), pages 625-636. dejiao zhang, feng nan, xiaokai wei, shang-wen li, henghui zhu, kathleen mckeown, ramesh nallapati, andrew o. arnold, and bing xiang. 2021. supporting clustering with contrastive learning. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 5419-5430, online. association for computational linguistics.