Are Neural Topic Models Broken? (2022)

Abstract: Recently, the relationship between automated and human evaluation of topic models has been called into question. Method developers have staked the efficacy of new topic model variants on automated measures, and their failure to approximate human preferences places these models on uncertain ground. Moreover, existing evaluation paradigms are often divorced from real-world use.Motivated by content analysis as a dominant real-world use case for topic modeling, we analyze two related aspects of topic models that affect their effectiveness and trustworthiness in practice for that purpose: the stability of their estimates and the extent to which the model’s discovered categories align with human-determined categories in the data. We find that neural topic models fare worse in both respects compared to an established classical method. We take a step toward addressing both issues in tandem by demonstrating that a straightforward ensembling method can reliably outperform the members of the ensemble.

## introduction topic models provide an unsupervised way both to discover implicit categories in text corpora, and to estimate the extent to which any given category applies to a specific text item. as such, topic modeling can be viewed as an automated variety of content analysis for text: those two capabilities directly correspond to the practice of developing an emergent coding system via examination of a text collection, and then coding the text units in the collection @xcite . this form of content analysis is a dominant use case for topic models and therefore it is our focus here. explicitly identifying topic models as a tool for content analysis allows us to characterize what makes topic models good: we can measure the extent to which a model achieves the goals of content analysis. this careful consideration of the criteria for "good" topic models is essential because recent results have challenged the validity of the prevailing model evaluation paradigm @xcite @xcite . in particular, hoyle et al. identified a validation gap in the automatic evaluation of topic coherence: metrics like the widely-used normalized pointwise mutual information (npmi) were never validated using human experimentation for the newer neural models once they emerged, and the authors demonstrated that such metrics exaggerate differences between models relative to human judgments. given that the majority of claimed advances in topic modeling are predicated on these metrics (per hoyle et al.'s meta-analysis), it would appear that much of the topic model development literature now rests on uncertain ground. doogan and buntine also challenged the validity of current automated evaluation measures, and highlighted the disconnect between these measures and use of topic models in real-world settings. in this paper, we begin with the needs of content analysis, and we use those needs to argue for specific choices of how to measure topic model performance. we then report on comprehensive experimentation using two english-language datasets, four neural topic models that are representative of the current state of the art, and classical lda with gibbs sampling as implemented in mallet @xcite . the results indicate that mal-let is a more reliable choice than the more recent neural models from a content analysis perspective. taking a step toward addressing these issues, we use a straightforward ensemble method that combines the output of models across runs, which reliably yields better results than the usual practice of running a single model. to summarize our argument and contributions: we release all code and data. 1 2 what makes a topic model "good"? in considering how to characterize a topic model that works well, we focus on text content analysis as a dominant use case for topic modeling. ## experiments having argued that topic models should be subject to evaluations designed with real-world uses in mind, and having motivated specific ways to operationalize evaluative measurements based on criteria that matter for text content analysis, we evaluate nominally "state-of-the-art" topic models to understand how well they perform relative to those criteria. ## results recall that measuring stability is motivated by intracoder reliability in content analysis: producing the same result every time increases confidence that the analysis reflects actual latent structure in the data. mallet is significantly more stable than other models across the vast majority of contexts, often by a large margin ( recall also that alignment is motivated by intercoder reliability in content analysis: is the model, in the role of analyst, agreeing with human-derived categorization for the data? mallet shows strong consistency in providing the numerically best alignment with human categorization across datasets ( results are in appendix a.1. unsurprisingly, fixing "good default" hyperparameters for the neural models improves their stability and alignment. in particular, d-vae has competitive alignment metrics in the |v | = 5k case, although it is hampered by its relatively poor stability. mallet's stability is marginally affected: while it is no longer as consistently dominant, it remains more stable and better-aligned in the majority of contexts. ## a close reading of model stability it is immediately clear that even the nearest topic for d-vae has fewer words in common with the base topic, compared to mallet. and as distance increases, the top words for mallet stay consistent, whereas those for d-vae change dramatically, even if they relate to the same weather concept. ## ensembling estimates we have highlighted lack of stability as a serious problem for neural topic models, but neural models can also have desirable properties. how can we increase the odds of obtaining a good neural topic model in the face of extreme variation? the distance metrics we use to measure instability offer one solution: clustering to aggregate similar estimates over runs to form an ensemble. we adopt an approach similar to prior work @xcite , going further by accounting for the document-word estimates θ and by evaluating ensembles' alignment against human categorization. specifically, we concatenate run estimates over the m runs @xmath3, where each row in the concatenated matrix is a topic. we then compute pairwise distances between topics, d( b) and d( θ), and cluster based on a linear interpolation of the two distances, λd( b) + (1λ)d( θ), where λ is a hyperparameter. the estimate of each topic k from each run i, ⟨θ to evaluate this method, we compare the alignment score of the ensemble ( §2.4.2) combining the @xmath4, versus the alignment score of each individual member run. we do so across each of the 400 contexts (model, dataset, k, high versus low label granularity, and metric). figure 1 illustrates a summary of results for the purity alignment metric on the wikipedia dataset. across the full range of our experimentation, the ensemble improves on the median member in 97% of all the contexts, and it is always better than the worst member (full results in appendix a.5). ## conclusions a tool can be considered broken when it doesn't work well for its intended use. in this paper we have focused on a widespread use case for topic models, their application in text content analysis; we have carefully motivated criteria for measuring the extent to which a topic model is serving those needs; and we have demonstrated through comprehensive and replicable experimentation that, when measured on those criteria, recent and representative neural topic models fail to improve on the classical implementation of lda in mallet. in particular, mallet is much more stable, reducing concerns from the content analysis perspective that different runs could yield very different codesets. equally important, across the vast majority of contexts, its discovered categories are reliable as measured via alignment with ground-truth human categories. for people seeking to use topic modeling in content analysis, therefore, mallet may still be the best available tool for the job. that said, there are still good reasons to investigate neural topic models. foremost among these is the fact that they can benefit from pretraining on vast, general samples of language (e.g. @xcite @xcite . neural realizations of topic models can also be integrated smoothly for joint modeling within larger neural architectures (e.g. @xcite @xcite , and hold the promise of being more straightforward to use multilingually (e.g. @xcite @xcite or multimodally (e.g. @xcite . 14 we therefore introduced one possible way to address the shortcomings we identified using a straightforward ensemble technique. perhaps the most important take-away we would suggest is that development of new topic modelsindeed, of all nlp models-should be done with use cases firmly in mind. some models are enabling technologies, without a direct user-facing purpose, and others are intended to produce results directly for human consumption. but whatever the goal, the driving question for methodological development and evaluation should not be how to demonstrate an improvement in "state of the art", it should be why the model is being created in the first place and what measurements will demonstrate improved performance for that intended purpose.