An Empirical Analysis of Topic Models: Uncovering the Relationships between Hyperparameters, Document Length and Performance Measures (2021)

Abstract: Neural Topic Models are recent neural models that aim at extracting the main themes from a collection of documents. The comparison of these models is usually limited because the hyperparameters are held fixed. In this paper, we present an empirical analysis and comparison of Neural Topic Models by finding the optimal hyperparameters of each model for four different performance measures adopting a single-objective Bayesian optimization. This allows us to determine the robustness of a topic model for several evaluation metrics. We also empirically show the effect of the length of the documents on different optimized metrics and discover which evaluation metrics are in conflict or agreement with each other.

## introduction topic models @xcite are probabilistic generative models that aim at identifying the underlying themes, or topics, in a collection of documents. although they are used in a vast range of applications, from text exploratory purposes to information retrieval tasks @xcite , most of the investigations disregard the elements that influence the results generated by the models and, in particular, what is the effect on their performance. several works explore topic modeling over a range of different models, topics, and measures, but usually focus on classical topic models @xcite , e.g. latent dirichlet allocation (lda) @xcite or non-negative matrix factorization (nmf) @xcite , and solely on a single evaluation measure @xcite . doan and hoang recently made an effort to benchmark neural topic models, however, the authors seem to disregard the importance of the hyperparameter selection. in fact, the evaluations of topic models are usually limited to the comparison of models whose hyperparameters are fixed. yet, the hyperparameters that control the models can have a great impact on their performance. therefore, fixing them prevents the researchers from discovering the best topic model on a given dataset. in the latest years, neural topic models (ntm) @xcite @xcite have gained popularity, due to their flexibility and scalability. the problem of finding the best hyperparameter configuration has become even more compelling, since topic models based on neural networks are usually controlled by a high number of hyperparameters. in this paper, we perform an empirical analysis of recent ntms by optimizing the hyperparameters of the models with respect to different metrics. we aim to investigate if there exists a potential relationship between hyperparameters, document length and performance measures, to finally understand under which conditions we can exploit at best the potentiality of each model. in particular, the following research questions have been addressed: rq1: to what extent are neural topic models robust across different evaluation metrics? rq2: does the document length affect the neural topic models on different performance measures? rq3: does the optimization of a model's hyperparameters for a given performance metric imply good performance on other measures? to this purpose, we use bayesian optimization (bo) @xcite , a wellknown and efficient strategy for hyperparameter tuning, to determine the optimal hyperparameter settings for four different evaluation metrics of five state-of-the-art ntms. the hyperparameter optimization allows us to guarantee a fair comparison between the models and investigate their behavior with different hyperparameter settings. ## methodology in this work, we conduct an empirical comparison of different state-of-the-art ntms. we adopt a single-objective bayesian optimization approach, using the comparative framework topic modeling octis @xcite , to optimize the hyperparameters of five different topic models with respect to four different evaluation metrics. each metric investigates a different aspect of a model. bayesian optimization @xcite ) is a sequential model-based optimization (smbo) strategy that allows us to optimize all the hyperparameters by treating the topic model as a black-box function. the model is in fact viewed just in terms of its input (the hyperparameters) and its output (the distribution of the topics over the vocabulary and the topic distribution for each document). bo uses the model's configurations evaluated so far to approximate the value of the performance metrics with respect to the model's hyperparameters and then selects a new promising configuration to evaluate. the two key components are the probabilistic surrogate model aimed at approximating the performance metrics to optimize, and the acquisition function that uses the mean of the surrogate model and the confidence (i.e. its standard deviation) to select the next configuration. optimizing a model's hyperparameters not only allows us to investigate the robustness of a model over different evaluation metrics (rq1), but we can also investigate the performance of the optimized evaluation metric on datasets with different features (rq2) and the relationship between the optimized evaluation metric and the other metrics (rq3). ## conclusions and future work in this paper, we presented an empirical analysis of neural topic models to determine the relationship between hyperparameters, document length and performance measures. three main research questions have been addressed for understanding under which conditions such topic models could work for guaranteeing their best performance. the main findings could help both practitioners on tuning the models according to their objectives and researchers to explore the role of hyperparameters and document length with respect to any given performance measure. regarding the future work, the problem of hyperparameter optimization by considering multi-objective optimization @xcite will be addressed for understanding to which extent multiple metrics could be optimized according to the length of the documents and the hyperparameters of the models.