Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis (2024)

Abstract: Topic models are a popular tool for understanding text collections, but their evaluation has been a point of contention. Automated evaluation metrics such as coherence are often used, however, their validity has been questioned for neural topic models (NTMs) and can overlook a model’s benefits in real-world applications. To this end, we conduct the first evaluation of neural, supervised and classical topic models in an interactive task-based setting. We combine topic models with a classifier and test their ability to help humans conduct content analysis and document annotation. From simulated, real user and expert pilot studies, the Contextual Neural Topic Model does the best on cluster evaluation metrics and human evaluations; however, LDA is competitive with two other NTMs under our simulated experiment and user study results, contrary to what coherence scores suggest. We show that current automated metrics do not provide a complete picture of topic modeling capabilities, but the right choice of NTMs can be better than classical models on practical tasks.

## introduction establishing a label set to organize a collection of documents is a fundamental task in many fields such as social science, and, linguistics, education. for example, in the social sciences, grounded theory emphasizes structural coding as a framework for discovering similarities and differences in largescale experimental data and assigning meaning to it @xcite @xcite . such a process is difficult and time-consuming, partly because it requires a global understanding of the entire dataset, and local knowledge to accurately label individual documents. we emphasize that this strictly more general than document classification: classification presumes a priori a label set; while we will use classifiers in our method, we first need a user's help to determine the label set and the training data. topic modeling @xcite has emerged as a popular tool to help with the coding process to discover the label set (section 2.4). these models treat documents as admixtures of latent topics, each represented by a distribution over words. the most popular topic model, latent dirichlet allocation (lda) @xcite has over 40,000 citations with numerous extensions and variants @xcite . previously, active learning with topic overviews @xcite demonstrated that combining lda with an active learning classifier could help people create label sets more efficiently. after topic models provide a global overview of the data, exposing the broad themes of the corpus, active learning selects documents that direct the annotator's attention to topically distinct examples to label. together, these two ingredients train a classifier to automatically label the documents more efficiently. however, a gap remains in the literature, given recent advancements in topic modeling. neural topic models (ntm), which use continuous text embeddings to capture contextual and semantic relationships in high-dimensional data, have gained prominence, besting classical probabilistic topic models on automatic evaluation metrics such as coherence @xcite . however, automated evaluation metrics have been called into question; @xcite show they do not necessarily correlate with human ratings on topic model outputs and call for task-centered evaluations, such as helping users analyze content. we aim to fill this gap, and evaluate the effectiveness of neural, supervised, and classical topic models to help social scientists with content analysis and label set creation. we do this by taking the starting point of alto-classicial topic models applied to this problem-and probe "deeper" to create topic-enabled neural organization and recommendations (tenor), an interactive tool that supports various topic models with active learning to speed up the process of content analysis. 1 we conduct synthetic experiment on lda, supervise lda and three ntms with followup user study and expert user study and show that the choice of contextualized topic model @xcite (ctm) helps users create higher quality label sets than using classical lda, as measured by both cluster metrics (section 4.3) and user ratings. however, lda is still competitive or better when compared with two other popular ntms. thoughtfully using topic models as part of a larger system with human interactions gives a more comprehensive evaluation and understanding of their real-world usage (section 4.5). ## background manually sorting thousands of documents to establish a label set to create is mentally challenging and time-consuming. @xcite compare grounded theory with topic modeling: although the two methods are from distinct fields, they produce similar insights on large-scale data. topic models cluster documents and extract meaningful themes and can help users induce labels. for content analysis, machine learning and nlp focus on developing ntms @xcite , because they win nearly every automatic coherence metric. however, most of the computational social science community remains focused on older probabilistic models @xcite . thus, we explore this open question: should we use classical or neural topic models for label induction and content analysis? one of the reasons that ntms might be better is that alto showed the benefits of active learning @xcite : start with a dataset with an undefined label set; users add labels to the set by going through individual documents (guided by topic overviews); once the users establish at least two distinct labels for the label set, a classifier trained on the labeled documents can point users to documents that are either challenging for the current label set or that might require new labels. one of the criticisms of ntms is that they are too granular and specific @xcite , but this may be a boon for label induction: it can find candidates for a new label. in addition to ignoring neural models (which had not reached maturity when alto was proposed), another lacuna of @xcite is that it ignores supervised topic models that can combine classification with topics. supervised topic models @xcite change as labels are added and can adapt-for instancewhen a user associates two labels with a topic. thus we evaluate neural and classical topic models that tasks humans with creating a label set and annotating a document collection, with the assistance of topic models and a text classifier on the a dataset of us congressional bills (adler and @xcite . we delve into specific topic models, active learning, and evaluation metrics for the rest of this section. ## user study we conduct a general user study and expert study to compare topic models in the rest of the paper. for the general user study, we compare settings (1) none, (2) lda, (3) slda, and (6) ctm since ctm is the best-performing neural model in the simulated experiment. we use the bills dataset to conduct a 60-user study with our interface, with 15 people in each group. our bills dataset's topics are accessible to lay annotators and allows us to quantitatively understand users' acclimation to the dataset as they explore the corpus. we then run a smaller, more qualitative followup expert study on an expert dataset, where the experts are familiar with the topics in the dataset with the best two models from our user study results. the goal of the expert study is to ensure that our user study results can generalize to experts with deeper knowledge of us federal policy. ## related work applications of topic models are important, as exemplified by previous work by fang et al @xcite , which addresses the human-centric applications of topic models. @xcite shows that interactive topic models have gained traction among social science researchers and data analysts. nevertheless, classical topic models dominate most applications in social science research @xcite . despite their theoretical advantages, this persistent preference for classical models underscores the need for comprehensive studies on the practical utility of ntms. as one of the most popular topic models, lda has been widely applied and tested in diverse fields from health @xcite to political opinion analysis @xcite , social media data analysis @xcite , etc. thus, lda has already proved itself as a useful tool for real applications. for supervised models, most work focuses on slda's power to predict response variables from text @xcite . few works have study whether the induced topics align with user intents such as labeling. using slda interactively for document recommendation and annotation is more intuitive and straightforward than using unsupervised classical lda. beyond connecting a single response variable to topic assignments, neural models offer even more flexibility and have over a hundred variants, but the evaluation of ntms is mainly based on topic coherence, topic diversity, and classification applications @xcite . the major framework of ntms are mostly sequential ntms, which leverages the architecture power of recurrent neural network (rnn); ntms with pre-trained language models, such as bert, that already learns the semantic relationship and association of words from a large corpus of texts. ntms have the advantage of producing higher automatic evaluation scores, and classification abilities, along with other more extensive applications that classical topic models cannot do, which includes texts generation @xcite , summarization @xcite . however, with the new popularity of ntms, to the best of our knowledge, there are still few works using ntms for social science due to their complex architecture and more computing resource demands. our work examines this gap to study the trade-off between using neural, supervised, or classical topic models. while some recent studies have compares ntm and lda with human analysis of the topic outputs, they still predominantly rely on auto-matic evaluation metrics, with limited emphasis on analyzing the quality of models from a human perspective or task-based utility of topic models @xcite . @xcite concludes that lda is better than ntm in metrics on coherence @xcite and classification @xcite . however, this conclusion is for non-english datasets. our research intends to bridge this gap by conducting an english-language topic model quality evaluation, incorporating human interaction to help content analysis. our approach differs from previous studies, which compares ntms and classical models' stability and alignment with stationary, pre-determined ground truth labels @xcite . in the former, lda was better; in the latter, lda was better than many ntms @xcite . however, @xcite 's approaches only evaluate topic models by analyzing human ratings on topic keywords with labels without any task applications. in contrast, for the tasks of content analysis and building a label set, the overly specific ntm keywords are actually helpful for people to come up with labels more easily than more general and dispersed keywords. while the overall topics may not look as "pretty" to a user, they are useful. ## conclusion we provide an interactive task-based evaluation of neural, supervised, and classical topic models, using the task of content analysis and label set creation. using ctm with an active learning classifier helps both expert and non-expert annotators produce higher quality label sets more quickly, according to cluster metrics and human ratings, validating that the right choice of ntms can be better than lda for content analysis. however, lda is still competitive with two other ntms, contrary to what coherence scores would suggest. we show that current automated metrics do not provide a complete picture of topic modeling capabilities, but the right choice of ntms can still be better than classical models on practical tasks. with the popularity of large language models (llms), future work can include exploring more effective ways to use tenor combined with llms for content analysis, where experts have a set of pre-defined research question and hypothesis, and use tenor to actively select documents to prompt an llm to build up a label set for the dataset quickly to answer their research questions and verify their research hypothesis.