TOMODAPI : A Topic Modeling API to Train, Use and Compare Topic Models (2020)

Abstract: From LDA to neural models, different topic modeling approaches have been proposed in the literature. However, their suitability and performance is not easy to compare, particularly when the algorithms are being used in the wild on heterogeneous datasets. In this paper, we introduce ToModAPI (TOpic MOdeling API), a wrapper library to easily train, evaluate and infer using different topic modeling algorithms through a unified interface. The library is extensible and can be used in Python environments or through a Web API.

## introduction the analysis of massive volumes of text is an extremely expensive activity when it relies on not-scalable manual approaches or crowdsourcing strategies. relevant tasks typically include textual document classification, document clustering, keywords and named entities extraction, language or sequence modeling, etc. in the literature, topic modeling and topic extraction, which enable to automatically recognise the main subject (or topic) in a text, have attracted a lot of interest. the predicted topics can be used for clustering documents, for improving named entity extraction @xcite , and for automatic recommendation of related documents @xcite . several topic modeling algorithms have been proposed. however, we argue that it is hard to compare and to choose the most appropriate one given a particular goal. furthermore, the algorithms are often evaluated on different datasets and different scoring metrics are used. in this work, we have selected some of the most popular topic modeling algorithms from the state of the art in order to integrate them in a common platform, which homogenises the interface methods and the evaluation metrics. the result is tomodapi foot_0 which allows to dynamically train, evaluate, perform inference on different models, and extract information from these models as well, making it possible to compare them using different metrics. the remaining of this paper is organised as follows. in section 2, we describe some related works and we detail some state-of-the-art topic modeling techniques. in section 3, we provide an overview of the evaluation metrics usually used. we introduce tomodapi in section 4. we then describe some datasets (section 5) that are used in training to perform a comparison of the topic models (section 6). finally, we give some conclusions and outline future work in section 7. ## related work aside from a few exceptions @xcite , most topic modeling works propose or apply unsupervised methods. instead of learning the mapping to a pre-defined set of topics (or labels), the goal of these methods consists in assigning training documents to n unknown topics, where n is a required parameter. usually, these models compute two distributions: a document-topic distribution which represents the probability of each document to belong to each topic, and a topic-word distribution which represents the probability of each topic to be represented by each word present in the documents. these distributions are used to predict (or infer) the topic of unseen documents. latent dirichlet allocation (lda) is a unsupervised statistical modeling approach @xcite that considers each document as a bag of words and creates a randomly assigned documenttopic and word-topic distribution. iterating over words in each document, the distributions are updated according to the probability that a document or a word belongs to a certain topic. the hierarchical dirichlet process (hdp) model @xcite is another statistical approach for clustering grouped data such as text documents. it considers each document as a group of words belonging with a certain probability to one or multiple components of a mixture model, i.e. the topics. both the probability measure for each document (distribution over the topics) and the base probability measure -which allows the sharing of clusters across documents -are drawn from dirichlet processes @xcite . differently from many other topic models, hdp infers the number of topics automatically. gibbs sampling for a dmm (gsdmm) applies the dirichlet multinomial mixture model for short text clustering @xcite . this algorithm works computing iteratively the probability that a document join a specific one of the n available clusters. this probability consist in two parts: 1) a part that promotes the clusters with more documents; 2) a part that advantages the movement of a document towards similar clusters, i.e. which contains a similar word-set. those two parts are controlled by the parameters α and β. the simplicity of gsdmm provides a fast convergence after some iterations. this algorithm consider the given number of clusters given as an upper bound and it might end up with a lower number of topics. from another perspective, it is somehow able to infer the optimal number of topics, given the upper bound. pre-trained word vectors such as word2vec @xcite or glove @xcite can help to enhance topic-word representations, as achieved by the latent feature topic models (lftm) @xcite . one of the lftm algorithms is latent feature lda (lf-lda), which extends the original lda algorithm by enriching the topic-word distribution with a latent feature component composed of pre-trained word vectors. in the same vein, the paragraph vector topic model (pvtm) @xcite ) uses doc2vec @xcite to generate document-level representations in a common embedding space. then, it fits a gaussian mixture model to cluster all the similar documents into a predetermined number of topics -i.e. the number of gmm components. topic modeling can also be performed via linearalgebraic methods. starting from the the high-dimensional term-document matrix, multiple approaches can be used to lower its dimensions. then, we consider every dimension in the lower-rank matrix as a latent topic. a straightforward application of this principle is the latent semantic indexing model (lsi) @xcite , which uses singular value decomposition as a means to approximate the term-document matrix (potentially mediated by tf-idf) into one with less rowseach one representing a latent semantic dimension in the data -and preserving the similarity structure among columns (terms). non-negative matrix factorisation (nmf) @xcite exploits the fact that the term-document matrix is non-negative, thus producing not only a denser representation of the term-document distribution through the matrix factorisation but guaranteeing that the membership of a document to each topic is represented by a positive coefficient. in recent years, neural network approaches for topic modeling have gained popularity giving birth to a family of neural topic models (ntm) @xcite . among those, doc2topic (d2t) foot_1 uses a neural network which separately computes n-dimensional embedding vectors for words and documents -with n equal to the number of topics, before computing the final output using a sigmoid activation. the distributions topic-word and document-topic are obtained by getting the final weights on the two embedding layers. another neural topic model, the contextualized topic model (ctm) @xcite uses sentence-bert (sbert) @xcite ) -a neural transformer language model designed to compute sentences representations efficiently -to generate a fixed-size embedding for each document to contextualise the usual bag of words representation. ctm enhances the neural-prodlda @xcite architecture with this contextual representation to significantly improve the coherence of the generated topics. previous works have tried to compare different topic models. a review of statistical topic modeling techniques is included in @xcite . a comparison and evaluation of lda and nmf using the coherence metric is proposed by @xcite . among the libraries for performing topic modeling, gensim is undoubtedly the most known one, providing implementations of several tools for the nlp field @xcite . focusing on topic modeling for short texts, stmm includes 11 different topic models, which can be trained and evaluated through command line @xcite . the topic modelling open source tool foot_2 exposes a web graphical user interface for training and evaluating topic models, lda being the only representative so far. the promoss topic modelling toolbox foot_3 provides a unified java command line interface for computing a topic model distribution using lda or the hierarchical multi-dirichlet process topic model (hmdp) @xcite . however, it does not allow to apply the computed model on unseen documents. ## metrics the evaluation of machine learning techniques often relies on accuracy scores computed comparing predicted results against a ground truth. in the case of unsupervised techniques like topic modeling, the ground truth is not always available. for this reason, in the literature, we can find: ## tomodapi: a topic modeling api we now introduce tomodapi, a python library which harmonises the interfaces of topic modeling algorithms. so far, 9 topic modeling algorithms have been integrated in the library ( for each algorithm, the following interface methods are exposed: the structure of the library, which relies on class inheritance, is easy to extend with the addition of new models. in addition to allowing the import in any python environment and use the library offline, it provides the possibility of automatically build a web api, in order to access to the different methods through http calls. the software is distributed under an open source license 7 . a demo of the web api is available at http://hyperted.eurecom.fr/topic . ## datasets and pre-trained models together with the library, we provide pre-trained models trained on two different datasets having different characteristics (20ng and afp). a common pre-processing is performed on the datasets before training, consisting of: the same pre-processing is also applied to the text before topic prediction. ## experiment and results we empirically evaluate the performances of the topic modeling algorithms described in section 2 on the two datasets presented in section 5 using the metrics detailed in section 3. for each algorithm, we trained two different models, respectively on 20ng and afp corpus. the number of topicswhen required by the algorithm -has been set to 20 and 7 when training on 20ng and afp, respectively, in order to mimic the original division in class labels of the corpora (except for gsdmm and hdp which infer the optimal number of topics). each model trained on either 20ng or afp is tested against the same dataset and the wikipedia dataset to compute each metric. the top ones in terms of coherence. when trained on afp, all topic models benefit of a bigger dataset; this results in generally higher scores and in different algorithms maximising specific metrics. we also consider the time taken by the different techniques for different tasks like training and getting prediction ( ## conclusions and future work in this paper, we introduced tomodapi, a library and a web api to easily train, test and evaluate topic models. 9 algorithms are already included in the library, while new ones will be added in future. other evaluation metrics for topic modeling have been proposed @xcite and will be included in the api for enabling a complete evaluation. among these, metrics based on word embeddings are gaining particular attention @xcite . for further exploiting the advantage of having a common interface, we will study ways to automatically tune each model's hyper-parameters such as the right number of topics, find an appropriate label for the computed topics, optimise and use the models in real world applications. finally, future work includes a deeper comparison of the models trained on different datasets.