Apples to Apples: A Systematic Evaluation of Topic Models (2021)

Abstract: From statistical to neural models, a wide variety of topic modelling algorithms have been proposed in the literature. However, because of the diversity of datasets and metrics, there have not been many efforts to systematically compare their performance on the same benchmarks and under the same conditions. In this paper, we present a selection of 9 topic modelling techniques from the state of the art reflecting a diversity of approaches to the task, an overview of the different metrics used to compare their performance, and the challenges of conducting such a comparison. We empirically evaluate the performance of these models on different settings reflecting a variety of real-life conditions in terms of dataset size, number of topics, and distribution of topics, following identical preprocessing and evaluation processes. Using both metrics that rely on the intrinsic characteristics of the dataset (different coherence metrics), as well as external knowledge (word embeddings and ground-truth topic labels), our experiments reveal several shortcomings regarding the common practices in topic models evaluation.

## introduction the automatic analysis of textual data has gained increasing levels of attention over the last few decades. the cost of manually analysing and annotating the ever-growing quantity of content created and shared on the web continues to be prohibitively expensive. topic modelling is an nlp task where, given a corpus of documents, the objective is to find the underlying meaningful clusters of documents (or topics) that are thematically coherent (use consistent and related vocabulary) and assign each document to one or more of these topics. as a text mining technique, it allows the analysis of big volumes of textual documents through clustering them into coherent sets addressing similar subjects (or topics), and labeling them using keywords that are understandable by the end-user. it has the advantage of not relying on any labeled data to achieve good results, as the training of topic models is done in an unsupervised matter. moreover, the resulting topics and representations can then be used to perform other nlp tasks such as trend prediction @xcite , text summarization @xcite , improving named entity recognition @xcite , and content recommendation @xcite . because of the unsupervised nature of the task, the evaluation of the quality of topic modelling techniques relies usually on metrics that do not require human annotation or ground-truth labels. most of the used "coherence" metrics -further detailed in section 3.1 -attempt to measure how much the resulting topics reflect some statistical characteristics of the original dataset and its word co-occurrences distribution. these metrics utilise different definitions of what a "coherent topic" is, and they only contingently agree with humans judgement @xcite . coupled with the different approaches for document preprocessing and the variety of used evaluation datasets, this complexity leads to several nuances in the evaluation process that are not widely acknowledged in the literature at large. thus, comparisons can be inconsistent and sometimes misleading. in this work, we selected a diverse array of topic modelling algorithms (probabilistic, algebraic, embedding-based and neural) from the literature and we provide a thorough comparison using a unified evaluation protocol. this protocol evaluates each topic model on several datasets, using a variety of metrics that range from intrinsic evaluation of the clustering quality to ones that assess the alignment between the extracted topics and the human-assigned labels. with this strategy, we aim to illustrate the inconsistency of these metrics when varying several subtle evaluation conditions. we analyse the results and we discuss the differences in performances across the different algorithms, datasets and parameters. the remainder of this paper is organised as follows. in section 2, we describe some related work, detailing some state-of-the-art topic modelling techniques. different metrics for evaluating topic models are introduced in section 3, while section 4 describes the datasets we use for this purpose. in section 5, we extensively analyse 9 topic models using coherence and ground truth related metrics. finally, we provide some conclusions in section 6. ## metrics the evaluation of machine learning techniques often relies on accuracy scores computed comparing predicted results against a ground truth. in the case of unsupervised techniques like topic modelling, the ground truth is not always available. for this reason, in the literature, we can find: ## datasets in this section, we introduce the datasets that we use in our experiments. the features of each dataset are reported in a common pre-processing is performed on the datasets before training, consisting of: ## experiment and results evaluating an unsupervised task such as topic modelling is inherently challenging, and despite the variety of metrics, it is still an open problem @xcite . while intrinsic metrics (coherence) try to measure the underlying quality of the topical clusters generated by each model, they do not always match with human judgement. two very coherent topics (according to the metric) can still fall under the same topic label for a human, and vice-versa. topic models aim to maximise the posterior probability of a document belonging to a coherent topic, regardless of how it maps to humanperceived categories. for instance, christianity and atheism can be both filed as two independent topics or one topic (religion) by a human annotator, and while neither arbitrary option is wrong, it constitutes a big difference to how we would evaluate the topic modelling algorithms. they have no means of inferring what humans find to be topically distinct beyond co-occurrence statistics, making the comparison to human-annotated labels (as a "gold standard") quite insufficient. because of these challenges, few works in the literature @xcite @xcite go beyond simple comparisons that only use one metric or dataset, eclipsing merits and shortcomings of the other methods. we attempt to provide a more thorough comparison using multiple evaluation datasets -varying in size, document length, number of topics, and label distribution -and metrics from the literature as a step towards a better understanding of the available options and their usability for different potential use-cases. ## conclusions and future work in this work, we empirically evaluated 9 topic modelling algorithms using different coherence and ground-truth-based metrics on 3 text corpora reflecting a variety of properties, using a common evaluation framework. the results reveal several differences between the trained models, which obtain better or worse performances depending on the evaluation setting. among these, lda proves to be the most consistent performer overall, while embedding-based models prove to be less prone to generating meaningless topics. the task of evaluating topic models remains a challenging one because of the inherent lack of a ground-truth, the subjectivity of what constitutes a "coherent topic", and the variety of settings wherein it is used. while every newly proposed topic model claims to improve on the existing state-of-the-art under some specific conditions, it is a worthwhile effort to revisit those claims and review them on a broader set of challenges and a unified pipeline, revealing their strengths and shortcomings. we also hope that by showing that no single metric can reflect the overall performance of any given topic model, we join a growing number of words drawing attention to the brittleness of most automatic metrics for topic models and the need of re-evaluating the standard practices of evaluation in the topic modelling literature. as an extension to this work, we intend to study how other factors such as language, preprocessing and dataset characteristics can influence the performance on the metrics, as well as develop a unified protocol for evaluation that can allow us to draw more interesting insights into how the different topic modelling approaches fare in real use cases and downstream applications.