LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models (2025)

Abstract: Topic modeling has been a widely used tool for unsupervised text analysis. However, comprehensive evaluations of a topic model remain challenging. Existing evaluation methods are either less comparable across different models (e.g., perplexity) or focus on only one specific aspect of a model (e.g., topic quality or document representation quality) at a time, which is insufficient to reflect the overall model performance. In this paper, we propose WALM (Word Agreement with Language Model), a new evaluation method for topic modeling that considers the semantic quality of document representations and topics in a joint manner, leveraging the power of Large Language Models (LLMs). With extensive experiments involving different types of topic models, WALM is shown to align with human judgment and can serve as a complementary evaluation method to the existing ones, bringing a new perspective to topic modeling. Our software package is available at https://github.com/Xiaohao-Yang/Topic_Model_Evaluation.

## introduction topic modeling @xcite , a popular unsupervised text analysis technique, has been applied to various domains, including information retrieval @xcite , marketing analysis @xcite , social media analysis @xcite , bioinformatics @xcite , and more. a topic model typically learns a set of global topics to interpret a text corpus and the topic proportion of a document as its semantic representation. although topic models have been time-tested for two decades, as an unsupervised technique, comprehensive evaluations of a topic model remain challenging @xcite . originally, topic models are implemented as probabilistic graphical models such as latent dirichlet allocation @xcite and many of its bayesian extensions (e.g., @xcite @xcite @xcite . for these models, it has been common practice to measure the log-likelihood or perplexity of a model on held-out test documents. while log-likelihood or perplexity provides a straightforward quantitative comparison between models, several issues still persist. since topic models are not primarily designed to predict words in documents but rather to learn semantically meaningful topics and interpretable document representations, these metrics fail to capture these aspects. furthermore, estimating the predictive probability is often intractable for bayesian models, and different papers may employ different sampling or approximation techniques @xcite . for recently proposed neural topic models (ntms) @xcite , the computation of log-likelihood is even more inconsistent. in addition to log-likelihood or perplexity, document representation quality and topic quality are evaluated separately. for document representation quality, downstream task performance is typically used as a metric, such as document classification @xcite , clustering @xcite , and retrieval @xcite . for topic quality, the ultimate evaluation method is human evaluation, which is time-consuming and expensive. thus, various automatic metrics have been proposed, such as topic coherence @xcite , which measures how semantically coherent the representative words in a topic are, and topic diversity @xcite , which measures how diverse discovered topics are. to comprehensively evaluate the performance of a topic model, one needs to report multiple metrics on both document representation and topic qualities. however, these metrics can be contradictory, e.g., a topic model with good topic quality may not preserve good quality on document representation, and vice versa. this discrepancy complicates the model selection process for topic models in practice. in this paper, we aim to develop a new evaluation approach for topic modeling that considers both the semantic quality of document representations and topics in a joint manner, leveraging the power of large language models (llms). our key idea is as follows: after being trained, a topic model can infer a document's distribution over topics and each topic is a distribution over vocabulary words. with these two distributions, a model can generate a set of ''topical'' words given a document, such as by looking at its representative topics and the representative words of each topic. the generation of the topical words takes both the topic distribution of a document and the word distributions of the topics into account, which captures the semantic summary of the document and is expected to align with the keywords identified by humans. given the high cost of human evaluation, we propose using llms as a proxy by employing appropriate prompts to generate keywords for the document, which are then compared with the topical words produced by a topic model. finally, to quantify the agreement between the words from the topic model and the llm, a series of walm (word agreement with language model) metrics are proposed. walm has the following appealing properties: to examine walm series metrics, we conduct extensive experiments using various popular topic models on different datasets, comparing them with other widely used topic model evaluation metrics. moreover, human evaluation is also conducted to demonstrate the alignment of walm with human judgment. ## related work as an unsupervised technique for uncovering hidden themes in text, evaluating topic models remains challenging. early evaluations of a topic model rely on the log-likelihood or perplexity of held-out documents @xcite , which measures how well the model predicts the words of documents. as the computation of predictive probability is often intractable for conventional bayesian topic models, various sampling or approximation techniques have been proposed @xcite . apart from the inconsistent estimation, held-out likelihood is regarded as not correlated with the interpretability of topics from a human perspective @xcite , prompting the direct evaluation of topics and document representation quality. as for the evaluation of topics, @xcite design the word and topic intrusion tasks for human annotators, where high-quality topics or document representations are those where annotators can easily identify the intruders. @xcite and @xcite evaluate topic coherence by direct ratings from human experts. although human judgment is commonly regarded as the gold standard, it is expensive and impractical for large-scale evaluation. automated evaluation of topic coherence is more practical, such as normalized pointwise mutual information (npmi) @xcite , which relies on the co-occurrence of the topic's top words in the reference corpus to measure topic coherence, with the underlying assumption that a large reference corpus such as wikipedia can capture prevalent language patterns. although they automate the evaluation of topics and strongly correlate with human judgment @xcite , counting word co-occurrence in a large reference corpus is still relatively expensive. moreover, coherence metrics can vary depending on the reference corpus, and there is no single ''right'' reference corpus that is suitable for all datasets @xcite . recent works propose leveraging word embeddings @xcite or contextualized embeddings @xcite for efficiently evaluating topic coherence, incorporating semantics from pre-trained embeddings. due to common posterior collapse issues @xcite in the growing field of neural topic models @xcite , recent works also consider topic diversity (dieng et al., 2020) during evaluation, which measures how distinct the top words of each topic are. as for the evaluation of document representation, early works focus on how well the topic proportion of a document represents the document content, assessed through a topic intrusion task by human annotators @xcite , which is further extended as automated metrics @xcite . recent topic models often use the topic proportions as document representations, the quality of which is commonly investigated through downstream tasks, including their use as features for document classification @xcite , clustering @xcite , and retrieval @xcite . recently, the generalization ability of topic models is investigated by evaluating their quality of document representations across different unseen corpora @xcite . in the era of llms @xcite @xcite , recent research has begun leveraging llms to evaluate topic models, such as using chatgpt 1 as a proxy for human annotators for word intrusion and topic rating tasks for evaluating topic coherence @xcite . the focus of these works is still on topic quality only. in this work, we propose new evaluation metrics for topic models, differing from previous works in the following ways: (1) unlike evaluations that focus on only sub-components of a topic model (i.e., topics or document representations), our evaluation metrics offer a joint approach to topic model evaluation, considering both topics and document representations together. (2) compared with log-likelihood or perplexity, which also evaluate based on documents, our evaluation metrics consider semantics from documents and align with the focus of topic modeling. (3) different from recent llm-based evaluations that use llms for topic quality evaluation, ours considers both topic quality and document representation quality and our use of llms is quite different from previous works. ## conclusion in this work, we propose walm for topic model evaluation, which takes both topic and document representation quality into account jointly. walm measures the agreement between the topical words generated by topic models and those from the llm for given documents. the topical words from the llm are obtained through keyword prompting or topic-aware keyword prompting, with the latter tending to capture higher-level information. to quantify the agreement between word sets, we propose different calculations, including overlap-based and embedding-based metrics. our experiments demonstrate that the walm series effectively reflect the capability of topic models to provide semantic summaries of documents. we show that walm metrics align with human judgment and can serve as an informative complementary method for topic model evaluation. we suggest that overlap-based metrics demonstrate better sensitivity handling, while embedding-based metrics show a smaller evaluation gap. a potential risk of using walm is that models chasing this metric only may be affected by the bias of llms. to mitigate the risk, we suggest using walm with other metrics together.