OCTIS : Comparing and Optimizing Topic models is Simple! (2021)

Abstract: In this paper, we present OCTIS, a framework for training, analyzing, and comparing Topic Models, whose optimal hyper-parameters are estimated using a Bayesian Optimization approach. The proposed solution integrates several state-of-the-art topic models and evaluation metrics. These metrics can be targeted as objective by the underlying optimization procedure to determine the best hyper-parameter configuration. OCTIS allows researchers and practitioners to have a fair comparison between topic models of interest, using several benchmark datasets and well-known evaluation metrics, to integrate novel algorithms, and to have an interactive visualization of the results for understanding the behavior of each model. The code is available at the following link: https://github.com/MIND-Lab/OCTIS .

## introduction topic models are promising statistical methods that aim to extract the hidden topics underlying a collection of documents. although researchers have proposed several models across the years @xcite , their evaluation and comparison is still a hard task. the evaluation of a topic model usually involves different datasets (with non-standard pre-processing) @xcite and several evaluation metrics @xcite @xcite . furthermore, topic models are usually compared by fixing their hyperparameters. however, choosing the optimal hyperparameter configuration for a given dataset and a given evaluation metric is fundamental to induce * corresponding author. current topic modeling frameworks @xcite @xcite typically focus on the release of topic modeling algorithms while ignoring one or more critical aspects of the topic modeling pipeline, such as preprocessing, evaluation, comparison of the models, and visualization. most importantly, they disregard the hyper-parameter selection. in this paper, we present octis (optimizing and comparing topic models is simple) foot_0 , a unified and open-source evaluation framework for training, analyzing, and comparing topic models, over several datasets and evaluation metrics. their optimal hyper-parameter configuration is determined according to a bayesian optimization (bo) strategy @xcite @xcite . in the following, we summarize the main contributions of the proposed framework: octis is an open-source evaluation framework for the comparison of a set of state-of-the-art topic models, that allows the user to optimize the models' hyper-parameters for a fair experimental comparison. the proposed framework follows an objectoriented paradigm, providing all the tools for running a topic modeling pipeline. the main functionalities of the proposed oc-tis are related to dataset pre-processing, training topic models, estimating evaluation metrics, hyperparameter optimization, and interactive web dashboard visualization. figure 1 summarizes the workflow involving the first four modules (the dashboard interacts with all of them). the framework can be used both as a python library and as a dashboard. the python library offers more advanced functionalities than the ones available in the dashboard. the modules that comprise the octis framework are detailed in the following sections. ## existing frameworks the existing topic modeling frameworks usually provide topic modeling algorithms, while disregarding other essential aspects of the whole topic modeling pipeline: pre-processing, evaluation, comparison, and visualization of the results and, most importantly, the hyper-parameter selection. in the fol-lowing, we outline the existing frameworks, highlighting their advantages and limitations. mallet @xcite and gensim 4 are the most known topic modeling libraries and include several classical topic models. they provide pre-processing methods and the estimation of the hyper-parameters using maximum likelihood estimation (mle) techniques. these libraries do not include the recently proposed neural topic models, and they just provide topic coherence metrics. sttm @xcite is a java library that provides a set of topic models that are specifically designed for short texts, providing several evaluation metrics. tomodapi @xcite ) is a python api that allows for training, inference, and evaluating different topic models, also including some of the most recent. however, it does not provide a method for finding the best hyper-parameter configuration of topic models. instead, a tool that allows for optimizing the hyper-parameter of a machine learning model is pycaret @xcite . however, it employs a grid-search technique to tune the hyper-parameters. this approach can be very time-consuming if the number of hyperparameters is high and the search space is huge @xcite . octis stands at the union of the features of the existing frameworks: we integrated both classical and recent neural topic models, providing preprocessing methods, evaluation metrics, and the possibility of optimizing the hyper-parameters. finally, a user-friendly graphical interface to launch one or more hyper-parameter optimization experiments on a given topic model and on a specific dataset has been provided. ## system usage octis has been designed to be used as a python library by advanced users, as well as a simple web dashboard by anyone. ## conclusions in this paper, we presented the framework octis for training, analyzing, and comparing topic models. the proposed framework is composed of a python library and a web dashboard and integrates several state-of-the-art topic models (both traditional and neural). these models can be trained by searching for their optimal hyperparameter configuration, for a given metric and dataset, exploiting a bo strategy. octis allows researchers to train existing models, integrate new training and inference algorithms, and fairly compare the topic models of interest. on the other hand, practitioners could use octis to boost the performance of topic models for their preferred downstream task or a wide range of practical applications, such as data exploratory analysis @xcite . regarding future work, octis could integrate a multi-objective optimization strategy to optimize multiple metrics in the same bo procedure @xcite . for example, this could allow a user to find an optimal hyper-parameter configuration for both topic coherence and document classification.