Revisiting Automated Topic Model Evaluation with Large Language Models (2023)

Abstract: Topic models help us make sense of large text collections. Automatically evaluating their output and determining the optimal number of topics are both longstanding challenges, with no effective automated solutions to date. This paper proposes using large language models (LLMs) for these tasks. We find that LLMs appropriately assess the resulting topics, correlating more strongly with human judgments than existing automated metrics. However, the setup of the evaluation task is crucial — LLMs perform better on coherence ratings of word sets than on intrustion detection. We find that LLMs can also assist us in guiding us towards a reasonable number of topics. In actual applications, topic models are typically used to answer a research question related to a collection of texts. We can incorporate this research question in the prompt to the LLM, which helps estimating the optimal number of topics.

## introduction topic models are, loosely put, an unsupervised dimensionality reduction technique that help organize document collections @xcite . a topic model summarizes a document collection with a small number of topics. a topic is a probability distribution over words or phrases. a topic t is interpretable through a representative set of words or phrases defining the topic, denoted w t . 1 each document can, in turn, be represented as a distribution over topics. for each topic, we can retrieve a representative document collection by sorting documents across topic distributions. we denote this set of documents for topic t as d t . because of their ability to organize large collections of texts, topic models are widely used in the social sciences, digital humanities, and other disciplines to analyze large corpora @xcite @xcite . interpretability makes topic models useful, but human interpretation is complex and notoriously difficult to approximate @xcite . automated topic coherence metrics do not correlate well with human judgments, often overstating differences between models @xcite . without the guidance of an automated metric, the number of topics, an important hyperparameter, is usually derived manually: practitioners fit various topic models, inspect the resulting topics, and select the configuration which works best for the intended use case @xcite . this is a non-replicable and time-consuming process, requiring expensive expert labor. recent nlp research explores whether large language models (llms) can perform automatic annotations; e.g., to assess text quality @xcite @xcite . here, we investigate whether llms can automatically assess the coherence of topic modeling output and conclude that: we use llms for two established topic coherence evaluation tasks and find that their judgment strongly correlates with humans on one of these tasks. similar to recent findings, we find that coherent topic word sets w t do not necessarily imply an optimal categorization of the document collection @xcite . instead, we automatically assign a label to each document in a d t and choose the configuration with the purest assigned labels. this solution correlates well with an underlying ground truth. thus, llms can help find good numbers of topics for a text collection, as we show in three case studies. ## topic model evaluation most topic model evaluations focus on the coherence of w t , the most probable words from the topic-word distribution @xcite . coherence itself can be thought of as whether the top words elicit a distinct concept in the reader @xcite . to complicate matters, human evaluation of topic models can be done in diverse ways. e.g., we can ask humans to directly rate topic coherence, for example, on a 1-3 scale @xcite @xcite . we can also add an unrelated intruder word to the list of top words, which human annotators are asked to identify. the intuition is that intruder words are easily identified within coherent and self-contained topics, but hard to identify for incoherent or not self-contained topics @xcite . high human accuracy on this task is thus a good proxy for high topic coherence. see both in example 1. intrusion detection task water area river park miles game horses horse breed hindu coins silver rating task health hospital medicare welfare insure 3 horses zurich race dog canal 1 example 1: two examples of intrusion detection (select outlier) and topic rating tasks (rate overall coherence). each example is on separate row. although many automated metrics exist @xcite @xcite , normalized pointwise mutual information @xcite is the most prevalent when evaluating novel methods @xcite . informally, npmi is larger if two words co-occur together regularly in a reference corpus. another popular metric, c v , is a combination of npmi and other measures and is also popular @xcite . see the formula definitions in appendix d. despite their popular use, these metrics correlate poorly with human evaluations @xcite . in this work, we let llms perform the rating and intrusion detection tasks for topic model evaluation foot_1 and propose llm scores as a novel automated metric. similar work by @xcite is carried contemporaneously. llms have already been used to rank machine translations and generated text @xcite @xcite and have also been shown to perform on par with crowdworkers @xcite . ## llm and coherence first, we show that large language models can assess the quality of topics generated by different topic modeling algorithms. we use existing topic modeling output annotated by humans @xcite . 3 this data consists of 300 topics, produced by three different topic modeling algorithms on two datasets: nytimes @xcite and wikitext @xcite . for each of the 300 topics, there are 15 individual human annotations for the topic word relatedness (on 1-3 scale), and 26 individual annotations for whether a crowd-worker correctly detected an intruder word. we replicate both tasks, prompting llms instead of human annotators. see prompt 1 for prompt excerpts, and appendix a for full details. intruder detection prompt. system prompt: [...] select which word is the least related to all other words. if multiple words do not fit, choose the word that is most out of place. [...] user prompt: water, area, river, park, miles, game rating task prompt. system prompt: [...] please rate how related the following words are to each other on a scale from 1 to 3 ("1" = not very related, "2" = moderately related, "3" = very related). [...] user prompt: lake, park, river, land, years, feet, ice, miles, water, area prompt 1: llm prompts for assessing topic coherence. we compute the spearman correlation between the llm answer and the human assessment of the topics and show results in baseline metrics. for npmi and c v , we report the best correlation by @xcite . these metrics depend on the reference corpus and other hyperparameters and we always report the best value. @xcite find no single best setting for these automated metrics and therefore this comparison makes the baseline inadequately strong. ## determining the number of topics topic models require specifying the number of topics. practitioners usually run models multiple times with different numbers of topics (denoted by k). after manual inspection, the model which seems most suited for a research question is chosen. @xcite review 189 articles about topic modeling and find that common use cases are exploratory and descriptive studies for which no single best number of topics exists. however, the most prevalent use case is to isolate semantically similar documents belonging to topics of interest. for this, doogan and buntine (2021) challenge the focus on only evaluating w t , and suggest an analysis of d t as well. if we are interested in organizing a collection, then we would expect the top documents in d t to receive the same topic labels. we provide an llm-based strategy to determine good number of topics for this use case: we let an llm assign labels to documents, and find that topic as-signments with greater label purity correlate with the ground-truth in three case studies. topics of interest might be a few broad topics such as politics or healthcare, or many specific topics, like municipal elections and maternity care. following recent efforts that use research questions to guide llm-based text analysis @xcite , we incorporate this desideratum in the llm prompt. we run collapsed gibbs-sampled lda (in mallet: @xcite on two text collections, with different numbers of topics (@xmath0, yielding 20 models per collection. to compare topic model estimates and ground-truth partitions, we experiment with a legislative bill summary dataset (from @xcite and wikitext @xcite , both annotated with ground-truth topic labels in different granularities. ## discussion in this work, we revisit automated topic model evaluation with the help of large language models. many automated evaluation metrics for topic models exist, however these metrics seem to not correlate strongly with human judgment on wordset analysis @xcite . instead, we find that an llm-based metric of coherent topic words correlates with human preferences, outperforming other metrics on the rating task. second, the number of topics k has to be defined before running a topic model, so practitioners run multiple models with different k. we investigate whether llms can guide us towards reasonable k for a collection and research question. we first note that the term optimal number of topics is vague and that such quantity does not exist without additional context. if our goal is to find a configuration which would result in coherent document sets for topics, our study supports evaluating d t instead of w t , as this correlates more strongly with the overlap between topic model and ground-truth assignment. this finding supports arguments made in doogan and buntine (2021) who challenge the focus on w t in topic model evaluation.