Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too! (2020)

Abstract: Topic models are a useful analysis tool to uncover the underlying themes within document collections. The dominant approach is to use probabilistic topic models that posit a generative story, but in this paper we propose an alternative way to obtain topics: clustering pre-trained word embeddings while incorporating document information for weighted clustering and reranking top words. We provide benchmarks for the combination of different word embeddings and clustering algorithms, and analyse their performance under dimensionality reduction with PCA. The best performing combination for our approach performs as well as classical topic models, but with lower runtime and computational complexity.

## introduction topic models are the standard approach for exploratory document analysis @xcite , which aims to uncover main themes and underlying narratives within a corpus. but in times of distributed and even contextualized embeddings, are they the only option? this work explores an alternative to topic modeling by casting 'key themes' or 'topics' as clusters of word types under the modern distributed representation learning paradigm: unsupervised pre-trained word embeddings provide a representation for each word type as a vector, allowing us to cluster them based on their distance in high-dimensional space. the goal of this work is not to strictly outperform, but rather to benchmark standard clustering of modern embedding methods against the classical approach of latent dirichlet allocation (lda; @xcite . we restrict our study to influential embedding methods and focus on centroid-based clustering algorithms as they provide a natural way to obtain the top words in each cluster based on distance from the cluster center. 1 aside from reporting the best performing combination of word embeddings and clustering algorithm, we are also interested in whether there are consistent patterns: embeddings which perform consistently well across clustering algorithms might be good representations for unsupervised document analysis, clustering algorithms that perform consistently well are more likely to generalize to future word embedding methods. to make our approach reliably work as well as lda, we incorporate corpus frequency statistics directly into the clustering algorithm, and quantify the effects of two key methods, 1) weighting terms during clustering and 2) reranking terms for obtaining the top j representative words. our contributions are as follows: analyzing documents by clustering word embeddings is a natural idea-clustering has been used for readability assessment @xcite , argument mining @xcite , document classification and document clustering @xcite , inter alia. so far, however, clustering word embeddings has not seen much success for the purposes of topic modeling. while many modern efforts have attempted to incorporate word embeddings into the probabilistic lda framework @xcite @xcite @xcite @xcite , relatively little work has examined the feasibility of clustering embeddings directly. xie and xing ( 2013 in contrast, our proposed approach is straightforward to implement, feasible for regular length documents, requires no retraining of embeddings, and yields qualitatively and quantitatively convincing results. we focus on centroid based k-means (km), spherical k-means (sk), and k-medoids (kd) for hard clustering, and von mises-fisher models (vmfm) and gaussian mixture models (gmm) for soft clustering; as pre-trained embeddings we consider word2vec @xcite , glove @xcite , fasttext @xcite , spherical @xcite , elmo @xcite , and bert @xcite . ## methodology after preprocessing and extracting the vocabulary from our training documents, each word type is converted to its embedding representation (averaging all of its tokens for contextualized embeddings; details in §5.3). following this we apply the various clustering algorithms on the entire training corpus vocabulary to obtain k clusters, using weighted ( §3.2) or unweighted word types. after the clustering algorithm has converged, we obtain the top j words ( §3.1) from each cluster for evaluation. note that one potential shortcoming of our approach is the possibility of outliers forming their own cluster, which we leave to future work. shows that after weighting (larger points have higher weight) a hopefully more representative cluster center is found. note that top words based on distance from the cluster center could still very well be low frequency word types, motivating reranking ( §3.3). ## computational complexity the complexity of km is o(tknm), and of gmm is o(tknm foot_2 ), for t iterations, 3 k clusters (topics), n word types (unique vocabulary), and m embedding dimensions. weighted variants have a oneoff cost of weight initialization, and contribute a constant factor when recalulculating the centroid during clustering. reranking has an additional o(n • log(n k )) factor, where n k is the average number of elements in a cluster. in contrast, lda via collapsed gibbs sampling has a complexity of o(tkn ), where n is the number of all tokens, so when n n, clustering methods can potentially achieve better performance-complexity tradeoffs. note that running elmo and bert over documents also requires iterating over all tokens, but only once, and not for every topic and iteration. ## experimental setup our implementation is freely available online. foot_3 ## results and discussion our main results are shown in ## conclusion we outlined a methodology for clustering word embeddings for unsupervised document analysis, and presented a systematic comparison of various influential embedding methods and clustering algorithms. our experiments suggest that pretrained word embeddings (both contextualized and non-contextualized), combined with tf-weighted k-means and tf-based reranking, provide a viable alternative to traditional topic modeling at lower complexity and runtime. the european chapter of the association for computational linguistics: volume 1, long papers, pages 991-1001. vivek kumar rangarajan sridhar. 2015. unsupervised topic modeling for short texts using distributed representations of words. in proceedings of the 1st workshop on vector space modeling for natural language processing, pages 192-200. felipe viegas, sérgio canuto, christian gomes, washington luiz, thierson rosa, sabir ribas, leonardo rocha, and marcos andré gonc ¸alves. 2019. cluwords: exploiting semantic word clustering representation for enhanced topic modeling. in proceedings of the twelfth acm international conference on web search and data mining, pages 753-761. pengtao xie and eric p xing. 2013. integrating document clustering and topic modeling. arxiv preprint arxiv:1309.6874. guangxu xun, yaliang li, wayne xin zhao, jing gao, and aidong zhang. 2017. a correlated topic model using word embeddings. in ijcai, pages 4207-4213. he zhao, lan du, and wray buntine. 2017. a word embeddings informed focused topic model. in asian conference on machine learning, pages 423-438.