Large-Scale Correlation Analysis of Automated Metrics for Topic Models (2023)

Abstract: Automated coherence metrics constitute an important and popular way to evaluate topic models. Previous works present a mixed picture of their presumed correlation with human judgement. In this paper, we conduct a large-scale correlation analysis of coherence metrics. We propose a novel sampling approach to mine topics for the purpose of metric evaluation, and conduct the analysis via three large corpora showing that certain automated coherence metrics are correlated. Moreover, we extend the analysis to measure topical differences between corpora. Lastly, we examine the reliability of human judgement by conducting an extensive user study, which is designed as an amalgamation of different proxy tasks to derive a finer insight into the human decision-making processes. Our findings reveal some correlation between automated coherence metrics and human judgement, especially for generic corpora.

## introduction topic modelling is an important tool in the analysis and exploration of text corpora in terms of their salient topics @xcite . to evaluate the effectiveness of topic models, the preponderance of topic modeling literature rely on automated coherence metrics. a key benefit is convenience, allowing researchers to sidestep expensive and timeconsuming user studies. the basis for this reliance is the assumption that the coherence metrics correlate with human judgement @xcite @xcite . the presumed correlation with human judgement should not be taken for granted. there are recent works that challenge the assumption. @xcite highlight the inconsistencies of automated coherence metrics via correlation analysis within each metric. in @xcite , they claimed some disagreement between human judgement and automated coherence metrics. we postulate that the reasons behind such a mixed picture could be the differences in the topic samples as well as the underlying corpora from which the statistics were derived, resulting in localised "biases" that affect the conclusions reached by respective studies. given their importance, we seek to conduct an extended analysis of automated coherence metrics on a larger scale than anything previously attempted. this study includes orders of magnitudes greater than the number of topics typically analysed, covering three large corpora, employing a comprehensive user study with extensive labels, across most of the widely used metrics. there is a strong motivation for quantity. given a vocabulary, a combinatorially large number of possible topics exist. if each topic is a vector of its scores on different metrics, the resulting curse of dimensionality @xcite necessitates a larger sample size. we argue that evaluating thousands of topics might not be sufficient, and a larger sample size is required to approximate a diverse distribution, where sampled topics is representative of the corpus and the metrics. we surmise that the previous practice of using topic models to generate topics could introduce a bias in the analysis. firstly, topic models vary in performance, @xcite compiled a lengthy list. there is also emerging debate on the performance between traditional and neural topic models @xcite . additionally, some neural models might be inconsistent, producing different topic sets in independent runs @xcite . conversely, topic model might be too stable and generate similar topics @xcite . to objectively evaluate whether the coherence metrics are usable, we propose to generate candidate topics independently of topic models. in this paper, our contributions are three-fold. first, we begin by analysing the inter-metric correlations (see section 4). we propose a novel approach to sample "topics" for the purpose of evaluating automated coherence metrics (see section 4.1). compared to prior works, we sample these topics free from topic model bias, and in a meaningful diverse manner. evaluated on three large corpora, we reaffirm that certain selected metrics do not contradict each other, and highlight the underestimated effects of ϵ (see section 4.2). second, we extend our analysis to investigate inter-corpora correlations (see section 5). we examine the understated differences of corpora statistics on the metrics by comparing the correlations across corpora. while such correlations do exist to some degree, the metrics are still dependent on each corpus. thus, any expectation that these metrics would correlate uniformly with human judgement on all possible corpora may be misplaced. finally, pivotal to any interpretability research, we design and conduct a user study, which is the keystone of our work (see section 6). compared to prior work, its design is more complex as we seek to benchmark human judgement at a finer granularity across different random user study groups (see section 6.1). we analyse the user study results via a few novel proxy measures, revealing that human judgement is nuanced and varies between individuals, metric correlation to human judgement is corpus-dependant, with the average participant being attuned to the generic corpora (see section 6.2). our implementation and releasable resources can be found here 1 , and we hope that it will enable convenient coherence evaluation of topic models and to further advance interpretability research. ## related work topic models. there are many approaches for topic modelling @xcite , from non-neural based @xcite ; @xcite , to many other neural-based methods, via autoencoders (kingma and welling, 2014) such as @xcite ; @xcite ; dieng et al. (2020); zhang and lauw (2020); @xcite , via graph neural networks @xcite @xcite , and hierarchical methods @xcite . a common factor is the use of automated coherence metrics to benchmark against baselines. we select several popular metrics for evaluation as listed in section 3. topic models are applied in downstream tasks @xcite @xcite . user studies in metric evaluation. mimno et al. ## preliminaries in this section, we define the automated coherence metrics that we will be using, and describe the corpora we use to obtain the word probabilities. ## examining inter-metric correlations intuitively, if two different metrics are to correlate with human judgement, we would expect the scores of these metrics to correlate. however, it is claimed in doogan and buntine (2021) that these metrics do not correlate well. for reasons described in section 1, we propose a new non-topic modelling approach to sample topics to evaluate these metrics. ## examining inter-corpus correlations a natural extension after inter-metrics comparison, is to compare metrics measured on different corpora. it is a common expectation that research works would employ multiple corpora, with the differences between corpora quantified superficially (such as in section 3.2). we propose an alternative approach to quantify the differences, at a topical level, using common topics measured using automated coherence metrics. if the corpora are thematically similar, we would expect a high correlation. analysis. using the common topics from the paired corpora, we conduct a correlation analysis on the scores measured on each corpus per metric. we conduct a control analysis on pairs of similar corpus differing in lemmatization, originating from the same documents, in ablations. while we know how lemmatization affects topic modelling @xcite , its effect on evaluation is unclear. we carried out two additional ablations simulating lemmatizing topics post-training. for the first ablation, we shortlist topics that contain at least one unlemmatized word, where if lemmatized, the lemmatized word can be found in the same unlemmatized corpus. we compare the correlation of the original and lemmatized topic, with their scores measured on the same unlemmatized corpus. their scores have a strong correlation (see ## user study previous works measure human judgement through simple evaluation tasks such as rating the coherence of a topic on a few-point ordinal scale @xcite , identifying the intruder word that introduced into the topic @xcite , or both @xcite . for word intrusion, the detection of outliers signals the cohesiveness of the topic, which is similar to rating topics on an ordinal scale. however for both tasks, qualitative gaps might exist. in word intrusion, study participants are restricted to just one outlier per topic, assuming perfect coding, it results in exponential drop in scoring, i.e. 100% detection for a perfect topic, 50% for a topic with a clear outlier, and so forth. for topic ratings, topics of differing qualities might get the same score, i.e. a perfect topic and a topic with a clear outlier might both get the same scores. additionally, while the decisions between human annotators might be equivalent, it is not evident if their thought processes are similar. the key reason for this line of inquiry stems from the observation that everyone is different in some aspects, such as knowledge, culture, and experiences. assuming our understanding of words is influenced by our prior beliefs, what and how we perceive similarity and coherence might differ from person to person. for these reasons, we decide to design a user study that combines both word intrusion and topic rating tasks but measured at a finer granularity such that we can quantify the decision-making process. users are tasked to cluster word-groups which indicate coherent and outlier word-groups. we then examine the relationships between automated coherence metrics and different proxy tasks derived from the user study. ## conclusion our large-scale analysis reaffirms that these automated coherence metrics are still meaningful. we are confident in using these metrics measured on generic corpus such as wiki, and specialised corpora, arxiv and pubmed, for nicher tasks. our user study empirically supports this conclusion, as our participants' collective response correlates well to metrics measured on wiki, albeit weaker but meaningful correlation on the specialized corpora. this work shows that popular automated coherence metrics, c npmi , c v , and c p , are alive and well, and works regardless of lemmatization. furthermore, we stress that the selection of the reference corpus is just as important as the selection of the metric, with wiki being the best reference corpus that correlates with human perception of coherence. moving forward, when evaluating for coherence aligned towards human interpretability, we recommend future topic models to be evaluated against wiki-variants. we also recommend calculating c v with ϵ = 0, to avoid the confusion from its contradiction of other metrics at ϵ = 1e-12.