Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures (2021)

Abstract: When developing topic models, a critical question that should be asked is: How well will this model work in an applied setting? Because standard performance evaluation of topic interpretability uses automated measures modeled on human evaluation tests that are dissimilar to applied usage, these models’ generalizability remains in question. In this paper, we probe the issue of validity in topic model evaluation and assess how informative coherence measures are for specialized collections used in an applied setting. Informed by the literature, we propose four understandings of interpretability. We evaluate these using a novel experimental framework reflective of varied applied settings, including human evaluations using open labeling, typical of applied research. These evaluations show that for some specialized collections, standard coherence measures may not inform the most appropriate topic model or the optimal number of topics, and current interpretability performance validation methods are challenged as a means to confirm model quality in the absence of ground truth data.

## introduction topic modeling has become a popular tool for applied research such as social media analysis, as it facilitates the exploration of large documentcollections and yields insights that would not be accessible by manual methods @xcite . however, social media data can be challenging to model as it is both sparse and noisy @xcite . this has resulted in increased demand for short-text topic models that can handle these challenges @xcite @xcite . topic word-sets, denoted t ws , are considered to be semantically related words that represent the latent component of the underlying topic's documentcollection, denoted t dc . meaning is derived from these topics through the interpretation of either the t ws @xcite , the corresponding t dc @xcite , or both @xcite . since meaning requires topics to be interpretable to humans, empirical assurance is needed to confirm a novel topic models' capacity to generate "semantically interpretable" topics, as well as a method to guide model selection and other parameters such as the number of topics, k. this is often achieved by calculating the coherence scores for t ws @xcite recent literature contradicts previous evaluations of some short-text topic models that claim superior interpretability @xcite @xcite . such rethinking flows from the fact there is no agreement on the best measure of interpretability @xcite and is compounded by the unclear relationship between human evaluation methodologies and automated coherence scores @xcite . finally, despite assurances of generalizability and applicability, topic model evaluations in machine learning are conducted in experimental settings that are not representative of typical applied use. this raises questions of whether coherence measures are suitably robust to measure topic interpretability and inform model selection in applied settings, particularly with challenging datasets like that of social media. advances in topic modeling for static documentcollections have produced non-parametric approaches such as hdp-lda, which employ sophisticated hierarchical priors that allow for different prior proportions @xcite . non-negative matrix factorization @xcite , the use of word embeddings, and neural network methods @xcite are a few of these other innovations. to support these advances, it is crucial to establish the robustness of topic modeling interpretability measures, especially given the growing trend towards evaluating topic models using coherence measures, often in the absence of perplexity or other predictive scores (?). additionally, increasingly sophisticated methods for automatic topic labeling have been developed. beginning with @xcite , this research relies on models which generate interpretable topics. while these advances enhance the technologies available to conduct applied research, they do not address the underlying question of whether topic interpretability can be adequately assessed using coherence measures. in this paper, we demonstrate a research gap in topic model evaluation methods in light of their growing use in specialized settings. previously declared state-of-the-art models are under-performing in applied settings @xcite , and little work has been done to improve application relevance @xcite . following the work of @xcite @xcite , this study examines whether coherence is a valid predictor of topic model interpretability when interpretability is defined as more than just the ability to label a t ws , and as the diversity of topic models, datasets and application tasks increases. earlier research has established a correlation between novel coherence measures and human ranking of interpretability, as measured by qualitative tests @xcite . however, since bounded experimental settings constrain these tests, they are unlikely to reliably and consistently indicate topic quality in applied research settings. as a result, we ask the following question: to what extent can we rely on current coherence measures as proxies for topic model interpretability in applied settings? this work has significant practical implications. it signals the need to re-develop interpretability measures and reappraise best-practice for validating and evaluating topic models and their applications. our research contributes the following: the remainder of this paper is organized as follows. section 2 provides a review of related work around the interpretability of topic models. section 3 describes five propositions that have informed the design of interpretable topic models and their evaluation measures. this is followed by a description of the experimental framework we designed to test these propositions. section 4 provides the results of these evaluations and section 5 contains a discussion of findings. ## background this section provides a brief overview of work related to interpretability evaluation, followed by a review of the challenges associated with coherence optimization for specialized contexts. ## evaluating interpretability we have generated five propositions about the relationship between coherence scores, human evaluation of topic models, and the different views of interpretability to explore the research question. we conduct five experiments to interrogate these propositions and re-evaluate how informative coherence measures are for topic interpretability. because we are evaluating existing coherence measures, we do not employ automatic topic labeling techniques. instead, we make use of human evaluation tasks that reflect those conducted in applied settings. proposition 1. if coherence scores are robust, they should correlate. the battery of coherence measures for evaluating novel topic models and automated labeling approaches are inconsistent across the literature. each new measure claims superior alignment to topic model interpretability. as these measures are evolutionary @xcite , and there is no convention for which measure should be used, particularly as a standard measure of qualitative performance @xcite @xcite , they are considered notionally interchangeable. thus, we would expect that there would be some correlation between these measures. however, previous studies have not considered the impact that the data type or model has on the coherence scores. particularly for non-parametric models, these issues may be compounded by how coherence measures are presented as an aggregate, e.g., the presentation of the top-n models. indeed, studies reporting multiple coherence measures have demonstrated inconsistencies at the model-level that are obscured during reporting @xcite . proposition 2. an interpretable topic is one that can be easily labeled. how easily a topic could be labeled has been evaluated on an ordinal scale where humans determined if they could hypothetically give a topic a label @xcite . however, humans are notoriously poor at estimating their performance, particularly when they are untrained and do not have feedback on their performance @xcite . thus, a rater's perception of whether they could complete a task is actually less informative than having them complete the task. ## results here we detail the results of our analysis of the five propositions about interpretability evaluation. ## discussion we repeated the work of @xcite , who demonstrated that when the top-ranked topics by c npmi are considered, metalda produces higher c npmi scores than lda. however, when c npmi was measured using alternative aggregate methods, we discovered that lda outperformed metalda. this is likely to be because the smaller topics in met-alda can be effectively ignored or scrapped, while in lda, all topics are of comparable size and are used by the model. other non-parametric topic models are belived to behave similarly. while metalda generated higher c npmi-abc scores than lda for all aggregates, it was highly dependent on dataset heterogeneity and the value of k. this should indicate that metalda is more adaptive to specialized language, an effect expected in other topic models supported by word embeddings. the comparative performance of coherence measures can vary significantly depending on the aggregate calculation method used and the way the data has been prepared. this latter point has been well established in the literature, most notably for twitter data @xcite , but is often overlooked when evaluating novel topic models. this is a cause for concern, given the growing reliance on coherence measures to select the optimal model or k in applied settings @xcite . propositions 2 and 3 addressed t ws interpretability. we have demonstrated the difference between comprehending a topic and providing a topic label that is both informative and reliable. however, coherence measures may not be informative of these qualities. propositions 4 and 5 addressed t dc interpretability. we have demonstrated that the ease of labeling a t dc and the alignment between t ws and t dc does not correlate with coherence measures. additionally, we identified several areas for future research into the use of diagnostic statistics in applied settings. we observed unexpected behaviors in the distributions of d ew and d tp after a comparative analysis between lda and the non-parametric model metalda, affecting the interpretability of both t ws and t dc . correlations between q dif /q aln and d ew /d tp for metalda, for example, indicate that these topic diagnostics could assist in evaluating t d c interpretability. ## conclusion we have shown that coherence measures can be unreliable for evaluating topic models for specialized collections like twitter data. we claim this is because the target of "interpretability" is ambiguous, compromising the validity of both automatic and human evaluation methods 9 . due to the advancements in topic models, coherence measures designed for older models and more general datasets may be incompatible with newer models and more specific datasets. our experiments show that non-parametric models, such as metalda, which employs embeddings to improve support for short-texts, behaves differently to lda for these performance and diagnostic measures. this is critical because recent research has focused on sophisticated deep neural topic models @xcite , which make tracing and predicting behaviors more challenging. abstractly, we may compare the use of coherence measures in topic modeling to the use of bleu in machine translation. both lack the finesse necessary for a complete evaluation, as is now the case with bleu @xcite . additionally, our study demonstrated that an examination of the t dc could provide greater insights into topic model behaviors and explained many of the observed problems. we argue for the representation of topics as a combination of thematically related t dc and t ws , and the further adoption of empirical evaluation using specialized datasets and consideration of t dc interpretability. to date, few papers have attempted this combination @xcite . however, we believe coherence measures and automated labeling techniques will continue to play a critical role in applied topic modeling. contextually relevant measures like c npmi-abc and topic diagnostics like d ew can be key indicators of interpretability. aside from the empirical evaluation of novel topic models, new automated labeling techniques, having proven themselves useful for labeling t tw , should be extended for t dc .