T opic GPT : A Prompt-based Topic Modeling Framework (2024)

Abstract: Topic modeling is a well-established technique for exploring text corpora. Conventional topic models (e.g., LDA) represent topics as bags of words that often require “reading the tea leaves” to interpret; additionally, they offer users minimal control over the formatting and specificity of resulting topics. To tackle these issues, we introduce TopicGPT, a prompt-based framework that uses large language models (LLMs) to uncover latent topics in a text collection. TopicGPT produces topics that align better with human categorizations compared to competing methods: it achieves a harmonic mean purity of 0.74 against human-annotated Wikipedia topics compared to 0.64 for the strongest baseline. Its topics are also more interpretable, dispensing with ambiguous bags of words in favor of topics with natural language labels and associated free-form descriptions. Moreover, the framework is highly adaptable, allowing users to specify constraints and modify topics without the need for model retraining. By streamlining access to high-quality and interpretable topics, TopicGPT represents a compelling, human-centered approach to topic modeling.

## introduction topic modeling is a commonly used technique for discovering latent thematic structures in extensive collections of text documents. traditional topic models such as latent dirichlet allocation @xcite represent documents as mixtures of topics, where each topic is a distribution over words. topics are often represented with their most probable words, but this representation can contain incoherent or unrelated words that make topics difficult for users to interpret @xcite . although some models enable users to interactively guide topics based on needs and domain knowledge @xcite , their usability is constrained by the bag-of-words topic format. to address these limitations, we introduce top-icgpt (figure 1 ), a human-centric approach to topic modeling that relies on prompting large language models to perform in-context topic generation ( §3.1) and assignment ( §3.2). first, we iteratively prompt an llm to generate new topics given a sample of documents from an input dataset and a list of previously generated topics. the resulting set of topics can then be refined to eliminate redundant and infrequent topics. finally, given a new document, an llm assigns it to one or more of the generated topics, also providing a quotation from the document to support its assignment. these quotations make the method easily verifiable, addressing some validity concerns plaguing traditional topic models. topicgpt produces higher-quality topics than competing approaches. topicgpt's topics and assignments align significantly more closely with human-annotated ground truth topics than those from lda, seededlda @xcite , and bertopic @xcite on two datasets: wikipedia articles @xcite and congressional bills (as processed by @xcite . we measure topical alignment using three external clustering metrics (harmonic mean purity, normalized mutual information, and adjusted rand index) and find that topicgpt improves substantially over baselines (e.g., absolute purity improves by 10 points over lda on wikipedia); furthermore, its topics are more semantically aligned with human-labeled topics (30.3% of topicgpt's topics are misaligned compared to 62.4% for lda on wikipedia). further analyses demonstrate the robustness of topicgpt's topic quality across various prompt and data settings. [document] instruction: merge topic pairs that are near duplicates. ## related work we designed topicgpt for the use case of automated content analysis, which is one of the primary applications of topic models @xcite . in this methodology, practitioners first engage in open coding of individual documents to establish initial labels. these initial codes are then carefully examined, reconfigured, and organized into a cohesive coding system @xcite @xcite . topic modeling for content analysis: traditional approaches to topic modeling, such as latent dirichlet allocation @xcite , are parameterized by topic-word and documenttopic distributions that can reveal latent thematic structures in a corpus. however, these representations are not straightforward to interpret @xcite @xcite , and subjective manual effort may result in issues of reliability and validity @xcite . our work follows previous research that aims to produce more interpretable topics in natural language @xcite @xcite . additionally, our design builds on seeded/anchored topic models @xcite @xcite , hierarchical topic models @xcite @xcite , as well as those that impose constraints on topics @xcite @xcite . llm-based content analysis: llms such as chatgpt have enabled new prompting and embedding-based approaches to analyzing text. researchers have used prompting techniques on these llms for related content analysis tasks, including text clustering @xcite @xcite , abstractive summarization @xcite , and deductive qualitative coding @xcite . prior work has explored topic modeling with contextualized embeddings from pre-trained models @xcite @xcite . recent work has also used llms to label @xcite and evaluate topics @xcite that are produced by an existing topic model. comparison to goalex: topicgpt most closely resembles goalex @xcite , but is tailored specifically for topic modeling. goalex focuses on clustering and corpus partitioning but not overall topic set organization, which is a fundamental need for content analysis. specifically, when refining topics, goalex only retains clusters such that each document is supported approximately once, which may remove topics that often appear together but have different meanings. instead, our framework refines topics based on their semantics to retain coherent, informative topics. additionally, where goalex assigns topics to documents individually, which scales poorly, topicgpt simultaneously provides all prompt topics to each document for efficient scaling. while goalex was evaluated only on cluster recovery, topicgpt is benchmarked on stability and alignment with groundtruth topics, demonstrating its usefulness as a content analysis tool beyond text clustering. ## methodology topicgpt consists of two main stages: topic generation ( §3.1) and topic assignment ( §3.2). figure 1 provides an illustrative overview of our framework. ## experiments our goal is to assess whether topicgpt's outputs align with human-coded ground truth topics, as well as to test its robustness to various settings. here, we describe our datasets, baseline methods, model configurations, and evaluation metrics. ## results our results demonstrate topicgpt's strong topical alignment with ground truth and robustness to variations in prompts and data. our human evaluation also show that the semantic content of topicgpt's generated topics (ignoring document assignment) is significantly more aligned with ground-truth topics on both datasets. furthermore, we explore the use of an open-source model for topic generation. ## future work topicgpt is designed for inductive content analysis, assuming the user has some prior familiarity with the dataset's content. however, the framework can also be applied to data exploration scenarios where the user is unfamiliar with the dataset and employs topic models to gain insights. future work could explore the potential of zero-shot prompting (without providing example topics or documents) with topicgpt for such exploratory use cases. appendix a presents a hierarchical extension of this framework. evaluating the performance of this extension against established hierarchical topic models through rigorous comparative studies will be a valuable next step. ## conclusion we introduce a prompt-based framework, top-icgpt, specifically designed for topic modeling. topicgpt addresses traditional topic models' interpretability and adaptability limitations by generating high-quality and descriptive topics. our results demonstrate that topicgpt outperforms baseline topic models in terms of topic alignment with ground truth labels while also showing robustness across different prompts and data subsets. we release our pipeline so that interested researchers and practitioners can try our framework.