Contextualized Topic Coherence Metrics (2024)

Abstract: This article proposes a new family of LLM-based topic coherence metrics called Contextualized Topic Coherence (CTC) and inspired by standard human topic evaluation methods. CTC metrics simulate human-centered coherence evaluation while maintaining the efficiency of other automated methods. We compare the performance of our CTC metrics and five other baseline metrics on seven topic models and show that CTC metrics better reflect human judgment, particularly for topics extracted from short text collections by avoiding highly scored topics that are meaningless to humans.

## introduction topic models are a family of text-mining algorithms that identify themes in a large corpus of text data @xcite . these models @xcite are widely used for exploratory data analysis with the aim of organizing, understanding, and summarizing large amounts of text data @xcite . numerous techniques, algorithms, and tools have been employed to develop a variety of topic models for different tasks and purposes @xcite including much recent work on neural topic models @xcite . however, due to their nature as unsupervised models, comparing topic outputs, hyperparameter settings, and overall model quality has traditionally been difficult @xcite . topic coherence (tc) metrics measure the interpretability of topics generated by topic models. these metrics are categorized into two classes: automated tc metrics and human-annotated tc metrics @xcite . automated tc metrics estimate the interpretability of topic models with respect to various factors such as cooccurrence or semantic similarity of topic words. on the other hand, human-annotated tc metrics are protocols for designing surveys that rate or score the interpretability of topic models. human judgment is often used to validate topic coherence metrics to provide an accurate assessment of the semantic coherence and meaningfulness of a given set of topics @xcite @xcite . while human-annotated tc metrics incorporate subjective human judgments and provide a more accurate and nuanced understanding of how well topic models are performing (e.g. in terms of their ability to capture the underlying themes in a text corpus), they are expensive, time-consuming, and require multiple human-subjects to avoid personal biases. on the other hand, automated metrics are more cost-effective than human-annotated methods, as they do not require the hiring and training of human annotators, which results in their ability to evaluate large amounts of data and iterate through many model comparisons. automated metrics are intended to align more closely with human judgment, providing a better measure of the interpretability of topic words. the risk of such approximations, however, is that they themselves become the target of optimization rather than the underlying property they were intended to measure. several recent works sug-gest that this has occurred especially in the context of neural topic models. @xcite argue that interpretability is ambiguous and conclude that current automated topic coherence metrics are unreliable for evaluating topic models in short-text data collections and may be incompatible with newer neural topic models. in a similar study, @xcite show that topics generated by neural models are often qualitatively distinct from traditional topic models while they receive higher scores from current automated topic coherence metrics. @xcite conclude that the validity of the results produced by fully automated evaluations, as currently practiced, is questionable, and they only help when human evaluations cannot be performed. @xcite in another recent work shows that neural topic models fail to improve on the traditional topic models such as gibbs lda @xcite and consider neural topic broken as they do not function well for their intended use. to address these problems, we introduce contextualized topic coherence (ctc) metrics which are a context-aware family of topic coherence metrics based on the pre-trained large language models (llm). taking advantage of llms elevates the understanding of language at a very sophisticated level incorporating its linguistic nuances, contexts, and relationships. ctc is much less susceptible to being fooled by meaningless topics that often receive high scores with traditional topic coherence metrics. ## automated topic coherence metrics topic coherence (tc) metrics measure the consistency of topic word representations (topic labels) to evaluate the interpretability and meaningfulness of a topic. most coherence measure are based on the analysis of topic word co-occurrence distributions within the model input documents. a high tc value indicates that the words in the topic labels are related and describe some semantic notion within a specific context or domain. @xcite claim that pointwise mutual information (pmi) based metrics achieve ratings which are highly correlated with human-annoted ratings. they define uci which measures the strength of the association between pairs of words based on their co-occurrence in a sliding window of length-l words. @xcite proposes umass, an asymmetric confirmation measure that estimates the coherence degree of topic labels by calculating the log ratio frequency of label word co-occurrences in the corpus of documents. umass counts the number of times a pair of words co-occur in a given corpus and compares this number to the expected number of co-occurrences of word pairs which are randomly distributed across the whole corpus. @xcite proposes context vector representations for topic words w to generate the frequency of word co-occurrences within windows @xmath0. they showed that npmi @xcite has a larger correlation with human topic ratings compared to uci and umass. additionally, npmi takes into account the fact that some words are more common than others and adjusts the frequency of individual words accordingly @xcite . while npmi is generally more sensitive to rare words and can handle small datasets, umass focuses on the fast computation of coherence scores over large corpora. @xcite showed that a smaller value of ϵ tends to yield better results than the default value of ϵ = 1 used in the original paper since it emphasizes more the word combinations that are completely unattested. @xcite proposes a unifying framework of coherence measures that can be freely combined to form a configuration space of coherence definitions, allowing their main elementary components to be combined in the context of coherence quantification. for example, they propose the c v metric, which uses a variation of npmi to compute topic coherence over a sliding window of size n and adds a weight γ to assign more strength to more related words. according to @xcite , the c v metric is more sensitive to noisy information and dirty data than c umass and c uci . nikolenko (2016) and @xcite propose the tc dwr metric based on the distributed word representations (dwr) @xcite which are better correlated to human judgment. similarly, @xcite presents a coherence measure based on grouping topic words into buckets and using singular value decomposition (svd) and integer linear programming-based optimization to create coherent word buckets from the generated embedding vectors. @xcite proposes several topic coherence metrics based on topic documents rather than topic words. the approach essentially extracts topic documents, vectorizes them using several methods such as word embedding aggregation, and computes a coherence score based on the document vectors. @xcite proposes an automated evaluation metric for local-level topic models by introducing a task designed to elicit human judgment and reflect token-level topic quality. @xcite investigate the evaluation of thematic coherence in microblog clusters and concludes that text generation metrics (tgms) proved most reliable, being less sensitive to time windows. similar to this work, @xcite explores the use of llms in evaluating topic models and determining the optimal number of topics in large text collections. ## contextualised topic coherence in this section we introduce contextualized topic coherence (ctc), a new family of topic coherence metrics that benefit from the recent development of large language models (llm). we present two approaches. the first approach uses llms to compute contextualized estimates of the pointwise mutual information (cpmi) between topic words. in the second approach, we use chatgpt (openai, 2022) to evaluate topic coherence by simulating to human-annotated evaluation methods. ditional pmi between words in context using a pre-trained language model. the cpmi between two words w i and w j in a sentence s is defined by the following equation: ## experiments in this section, we expect to observe that the baseline metrics (uci, umass, npmi, c v , dwr) rank topic models differently from ctc. we also expect ctc rankings favor interpretable topics and handle short text datasets more effectively than the baseline metrics @xcite . this implies that baseline metrics often yield high scores for incoherent topics, while conversely assigning low scores to well-interpretable topics. in contrast, ctc has a better model of language and can better evaluate topical similarity as it would appear to a human reader. therefore, we expect to see that baseline metrics and ctc would differ at extremes of highest or lowest coherency. ## human evaluation the goal of automated topic coherence metrics is to accurately approximate human judgment on topics without the need for expensive, timeconsuming studies that require multiple annotators to avoid bias. in this section we compare the proposed metric with a human evaluation data provided by @xcite . this data includes human evaluation scores (intrusion and ranking) for 50 topics generated by three topic models (gibbs lda @xcite , dvae @xcite , and etm (dieng et al., 2020)) applied on the (new york times) dataset. we evaluate the generated topics with ctc cpmi , ctc intrusion and ctc ranking , which are comparable to human intrusion and human ranking. as shown in we also can examine and compare different coherence metrics by analysing the topic words of high and low scoring topics. as shown in tables 5 and 7 , c v generates top topics which probably would not be chosen by a human. for example, the topic inc, 9mo, earns, otc, qtr, rev gets the highest score, even though it has little clear interpretability. on the other hand, ctc metrics score topics relative to their contextual relationship and are very close to human scores. for example, the topic film, theater, movie, play, director, movies receives the highest score by both ctc and human scoring. ## conclusion this paper introduces a new family of topic coherence metrics called contextualized topic coherence metrics (ctc) that benefits from the recent development of large language models (llm). ctc includes two approaches that are motivated to offer flexibility and accuracy in evaluating neural topic models under different circumstances. our results show that automated ctc outperforms the baseline metrics on large-scale datasets while semi-automated ctc outperforms the baseline metrics on short-text datasets. after a comprehensive comparison between recent neural topic models and dominant classical topic models, our results indicate that some neural topic models which optimize traditional topic coherence metrics, often receive high scores for topics that are overly sensitive to idiosyncrasies such as repeated text, and lack face validity. we show with our experiments that ctc is not susceptible to being deceived by these meaningless topics by leveraging the ability of llms to better model hu-man expectations for language and evaluate topics within and outside their contextual framework.